Volume 4 Number 1 Copyright 1998
Data Quality in the Practice of Consumer Product Management: Evidence from the Field
Barbara D. Klein, University of Michigan-Dearborn
Key Words: Organizations, Consumer Products, Databases, Validation, Detection, Correction
We found strong evidence that data stored in organizational databases have a significant number of errors. As computer databases continue to proliferate, the number of errors in stored data and the organizational impact of these errors are likely to increase. Two possible approaches to mitigating this problem are, first, to validate data as they are input or stored in databases, and, second, depend on users to detect and correct errors. Our continuing research program examining the efficacy of the second approach is underway. As part of this research program, we conducted interviews with consumer product managers. This paper discusses error detection incidents reported by consumer product managers, reports of data errors that were missed, possible payoffs for error detection, and perceptions about the base error rate.
There is strong evidence (e.g., Laudon, 1986; Morey, 1982; Redman, 1992) that data stored in organizational databases have a significant number of errors. Between one and ten percent of data items in critical organizational databases are estimated to be inaccurate (Laudon, 1986; Madnick and Wang, 1992; Morey, 1982; Redman, 1992). This estimate is based on the findings of several studies such as those by Laudon (1986) and Morey (1982). Inaccurate data have been reported in such diverse applications as a student loan database maintained by the U.S. Department of Education (Knight, 1992), in records maintained by the U.S. Department of Agriculture (1992), and in records maintained by credit reporting bureaus (1991), as reported by Newsweek and the Minneapolis Star Tribune.
As computerized databases continue to proliferate and as organizations become increasingly dependent upon these databases to support business processes and decision making, the number of errors in stored data and the organizational impact of these errors are likely to increase. Indeed, Mason (1986) argued over a decade ago that data quality will become an important issue facing MIS managers. Redman (1992) wrote that inaccurate and incomplete data may adversely affect the competitive success of an organization. For example, strategies such as "total quality management" may be difficult to implement if the data needed to support the decisions required by the strategy are not of adequate quality (Fox et al., 1993; Madnick and Wang, 1992; Redman, 1995). Errors in data may have a significant financial impact on organizations. For example, Dun & Bradstreet paid $350,000 to a construction company after incorrectly reporting that the company was bankrupt (Percy, 1986).
Two possible approaches to mitigating this problem are to validate data as they are input to or stored in databases (e.g., Morey) and to depend on users to detect and correct errors. A research program examining the efficacy of the second approach is underway. To date, several studies have been completed in this research stream. The first field research we conducted was in one business domain (actuarial science) and showed that at least some users of information systems detect errors in their organization's data (Klein, 1997). As the next step in our research, we conducted two laboratory experiments to examine the impact of base rate expectations, incentive structures, and error detection goals on error detection performance. We concluded that incentive structures and error detection goals affect error detection performance. We found no main effect for base rate expectations (Klein et al., 1997).
Next, a field study was conducted to link the results of the field research to practice in organizations. The overall objective of the field interview study was to increase our understanding of how users in different professional domains deal with data errors. We conducted interviews with consumer product managers, inventory managers, and municipal bond analysts. We presented a comparative analysis of the findings across these three professionals in Klein et al. (1996). We present a within-domain analysis of the findings of the consumer product manager interviews in this paper.
The remaining sections of our paper present (1) a review of prior research bearing directly on the question of the conditions under which individuals detect errors in data, (2) a theory of error detection, (3) the design of the field interview study, (4) the results of the study of the consumer product managers, and (5) a discussion of the results and conclusions.
In a broad sense, this investigation falls into the area of data and information quality. We can draw several general conclusions from existing data quality research. First, while no single definition of data quality has been accepted by researchers in this field, there is agreement that data accuracy, currency, and completeness are important areas of concern (Agmon and Ahituv, 1987; Davis and Olson, 1985; Fox et al., 1993; Huh et al., 1990; Madnick and Wang, 1992; Wand and Wang, 1994; Zmud, 1978). Second, while it is difficult to compare error rates across studies, rates substantially greater than zero have been found in all of the studies addressing the extent to which data errors exist in databases (Ham et al., 1985; Johnson et al., 1981; Knight, 1992; Laudon, 1986; Stone and Bublitz, 1984). Third, there is disagreement about the extent to which efforts to purge all errors from databases should be attempted. Some researchers propose methods designed to completely rid databases of errors (Janson, 1988; Svanks, 1988; Naus, 1975; Parsaye and Chignell, 1993), while others propose tools for determining how to best allocate limited resources to controlling the level of data errors (Ballou and Pazer, 1987; Ballou and Tayi, 1989; Ballou et al., 1987; Bowen, 1992; Paradice and Fuerst, 1991). Fourth, many researchers argue that users need not discard data containing errors. A variety of approaches for using imperfect data have been suggested (Ballou and Pazer, 1985; Ballou and Pazer, 1995; Ballou and Pazer, 1987; Bansal, 1993; Gaba and Winkler, 1992; Garfinkel et al., 1986; O'Leary, 1993; O'Neill and Vizine-Goetz, 1988). Finally, many statisticians, quality practitioners, and others involved in state-of-the-art manufacturing argue that error should be "designed out" of processes during R & D (Duncan, 1994).
This study examines an area of information quality that describes ways in which data users might modify their use of data if they become aware of errors. In general, the data quality literature argues that users are not very capable of finding errors in data nor of altering the way in which they use the data once errors are discovered. There is considerable evidence of poor user performance in detecting data errors. Over three decades ago, Davis et al. (1967) conducted a field experiment in which individuals were mailed banking confirmation statements with embedded errors. The individuals were asked to verify their account information, and approximately half failed to detect important errors. Laudon (1986) found that users of criminal information systems rarely detected errors in criminal records even though information provided to police departments by the FBI is accompanied by a warning stating that the user should verify that the information is accurate. Ricketts (1990) conducted a laboratory experiment in which over ninety percent of the subjects failed to detect a substantial data error in production planning reports. The failure of humans to detect errors in data is also assumed in much of the data quality literature where it is argued that (as in product manufacturing) resources should be devoted to the up-front improvement of the quality of data in organizational databases (e.g., Redman, 1992; Redman, 1995).
While previous research (e.g., Davis et al., 1967; Laudon, 1986; Ricketts, 1990) suggests that error detection is not a frequently occurring phenomenon, our field study of actuaries suggests that users in at least one professional domain detect data errors (Klein, 1997). Our research raises the question of whether error detection in organizational settings is a rare phenomenon occurring only under a limited set of conditions or occurs more frequently. We addressed this question by conducting field interviews for members of three professions who were selected based on expected variation along two theoretically derived dimensions: the actual base rate of errors in the domain and perceived payoffs for error detection in the domain. We present the results of one investigation into one domain - consumer product management - in this paper.
Underlying our research are theories of individual task performance and theories of accuracy in decision making.
Theories of individual task performance provide some general guidance for identifying conditions under which users detect errors in data. For example, experience seems to affect performance in general and may affect performance in error detection. One theory we could use is Campbell's (1990; Campbell and Pritchard, 1976) theory of individual task performance (depicted in Figure 1). This theory suggests that experience (e.g., Weber et al., 1993), knowledge, and effort (e.g., Payne, 1982; Payne et al., 1988) all affect error detection.
| Performance = f | (declarative knowledge |
| procedural knowledge and skills | |
| choice to expend effort | |
| choice of degree of effort to expend | |
| choice to persist)
|
Campbell's (1990) theory argues that performance on a particular component of a job is a function of an individual's declarative knowledge, procedural knowledge and skill and motivation. Campbell defines declarative knowledge as knowledge of the facts required to complete a task. Procedural knowledge is skill-based knowledge about how to perform a task. Declarative knowledge and procedural knowledge are claimed to be partially a function of education, training, and experience. Motivation is claimed to be a function of three choices: the choice to expend effort, the choice of the degree of effort to expend, and the choice to persist in task performance. Campbell suggests that experience and motivational influences can only affect job performance through changes in declarative knowledge, procedural knowledge and skill, or the three choices related to effort.
Our research views error detection as a specific component of some jobs. We define performance in our research as the successful or unsuccessful detection of an error in data. We argue here that variation in declarative knowledge and procedural knowledge affect error detection performance and that differences in expectations about the base rate of errors in data and assessments of the payoffs of error detection affect the rate at which errors are detected through the choices related to effort.
Several studies of expert performance suggest that significant amounts of experience are necessary for the development of expertise (e.g., Ericsson and Chase, 1982; Johnson et al., 1992a; Johnson et al., 1992b). This suggests that the actual number of errors that data users encounter will influence performance if they recognize problems and try to detect the errors. Recent research indicates that a large number of database errors have the potential, when adequate feedback occurs, to help build a concomitant knowledge base about the number and types of data error. Users working with data containing many errors may have more opportunities to develop the procedural knowledge and skills needed to detect errors than users working with data with a low base rate of errors. Therefore, users in domains with high error rates can develop effective strategies for error detection and remediation.
We believe that effort expended to detect errors may be a function, at least in part, of expectations about the base rate of errors in data and of user assessments about error detection payoffs. Campbell's (1990) theory of performance suggests that choices about the degree of effort to expend on error detection will influence performance. Our analysis of the data collected in our study of actuaries suggests that there are several factors that influence choices when users work with imperfect data. Our research suggests that data users make conscious and unconscious choices when they work with data. We discuss factors that influence data users' choices below.
Expectations about the Base Rate of Data Errors. As users expect more errors in data, greater effort may be devoted to inspecting and repairing data. Compared to users who expect a low base rate of errors in data, users who expect a high base rate of errors may expend more effort to detect and correct errors simply because they expect to detect more errors at any level of expended effort. There is evidence from our study of actuaries that their expectations about the base rate of errors in a source of data influenced their efforts to inspect and repair databases. For example, one user reported that she considers the base rate of errors in published mortality tables to be low and, therefore, she does not attempt to find errors in the tables. There is also evidence from the work of Weber et al. (1993) that at least some decision makers are sensitive to base rates.
Payoffs of Error Detection. We used a task performed by actuaries to illustrate how actuaries assess payoffs versus error detection performance. In one incident an actuary was using client-provided data. The objective was to determine whether an organization's financial reserves for its pension fund were sufficient. Such a judgement typically depends on an employee's pay rate and years of service. Imagine a specific case in which an actuary obtained this information, together with other personnel information for each employee of a company at the end of a year.
A user working with such a dataset might or might not suspect that the data in one of the fields is inaccurate (i.e., it is unlikely that a firm would hire an accountant at the age of 15). An actuary analyzing a pension fund might be likely to detect this error because it is quite material to the judgment about the sufficiency of the firm's pension reserves. On the other hand, a payroll manager reviewing the same dataset might be unlikely to find the error because errors in the Date of Birth and Number of Years of Service fields are not material to a firm's payroll.
Materiality. Thus, beliefs about the materiality of an error may influence the amount of effort expended to detect the error. Users may expend more effort to detect errors that they believe will have a significant impact on their calculations or decisions. We found evidence in our study of actuaries that the types of errors that actuaries try to detect and correct are directly proportional to the severity of the problems that bad data cause. For example, one actuary stated that there are some types of errors that he does not try to detect when pricing insurance because the errors would not have a significant impact on his calculations.
Incentives. An organization's incentives for detecting errors may also play an important role in users' assessments error detection payoffs. For example, an error in data that is successfully detected may generate additional work for the detector; and an incentive system that discourages the use of time to investigate and correct errors may create an environment in which many data errors go unnoticed.
Ease of Verification and Correction. How easily an error can be corrected may also influence the degree of effort expended on detecting the error. For example, it is possible that individuals won't try very hard to detect an error if (a) it is difficult to confirm that a suspected error is actually an error, or (b) if a confirmed error cannot be corrected. We have found that perceived payoffs for detecting errors may be quite low if detected errors cannot be corrected.
An underlying assumption of theories of effort and accuracy in decision making is that humans will devote no more mental resources or effort to a task than what is demanded by task requirements. This suggests that performance in error detection tasks may be sensitive to the specific performance requirements implicit in error detection payoffs. Research by Payne (1982; Payne et al., 1988; Johnson and Payne, 1985) demonstrated that task demands influence the selection of information processing strategies. Cryer et al. (1990) built on Payne's research to examine the influence of incentive schemes on information use and on task performance. Cryer et al. (1990) found that an incentive scheme that rewarded accuracy would lead to higher levels of task performance than an incentive scheme rewarding minimal effort. This finding supports the contention that error detection performance may be sensitive to variation in payoffs.
An initial interview with a person knowledgeable about consumer product management suggested that interviews of consumer product managers would provide an interesting contrast to our prior study of actuaries. Our a priori expectations stemming from this initial interview are summarized below.
Consumer product managers analyze data that are collected using scanners located at checkout counters in retail stores.Our initial interview indicated that the base rate of errors in this data is high and that the motivation of product managers to detect errors is low. The interview suggested that errors in scanning, errors in aggregation, and errors of classification (e.g., classification of the data by sales territory) occur in these data. We assumed that the effort to detect errors may be low in retail sales because there is little feedback about successful and unsuccessful error detection. A consumer product manager using scanner data to forecast product sales may never know if there is an undetected error because forecasts are not expected to exactly predict future sales. Ease of correction may also contribute to low motivation in the retail sales domain because it is difficult to verify that a suspected scanning error is actually an error. It is also difficult to correct a scanning error if one is verified. Our initial investigation also suggested that the materiality of errors in the retail sales domain may be low because some errors in scanner data are random in nature (because of the scanning process, for example), and the impact of these errors on analyses made using the data may not be significant.
Following our initial investigation, we interviewed five consumer product managers. To control for selection bias, potential interviewees were simply asked to participate in a study about the use of data in their work. The terms "error detection" and "data quality" were not used when recruiting subjects. Data were collected using a semi-structured interview. Several of the questions in the interview protocol are a variation on the critical incidents methodology developed by Flanagan (1954). These questions were designed to elicit descriptions of incidents when the interviewees successfully detected errors in data and other incidents when errors were missed.
We recorded and transcribed the interviews. We developed a coding scheme and coded the transcripts using this scheme.
Our analysis of consumer product manager interviews begins with two product managers describing error detection incidents. We next discuss strategies that the managers believe they use to find data errors. In the remaining sections we discuss instances when data errors were missed, payoff perceptions about error detection in the retail sales domain, perceptions about the base rate of errors in the domain, and evidence about how firmly consumer product managers believe that detecting data errors is a part of their employment responsibilities.
Two of the five consumer product managers reported detecting errors at least once. One of the consumer product managers reported an incident when a classification error occurred following a brand name modification in some markets.
"We had a product that was only sold in a four-pack. We then converted to a two-pack in about a quarter of the geography, but we kept the four-pack. At the same time, we changed the brand name. So what wound up happening is, when we were looking for a total, it was under the [Brand A] brand name, we changed it to [Brand B] and changed it to two-pack, and it was under four-pack. But we still had the four-pack [Brand B] product. We tried to create a new total that said, total [Brand A]-[Brand B]. The system ... counted it twice. So it counted it under [Brand A] and then it counted it under [Brand B]. And so even though your individual totals were correct, your macro total for the brand was way too high."
Another consumer product manager reported an incident in which the market share of a product was calculated incorrectly.
"I worked on a business called [Product Name] - it's a kid's beverage. And there was always debate within the brand and the company as to how to best calculate the share base. Some people said, "Oh, you share it against...all fruit beverages" or "you share it against just beverages that have fruit juice in it." I mean, there's all kinds of different ways to share that. Well, we had an error in reporting ... where they shared it against a base that we didn't want to use, and we thought all our shares had just collapsed. But they shared it against a bigger pie, and we were looking at a piece, and it was miscalculated."
The two consumer product managers who remembered occasions when they detected errors said that they do not detect errors very often. One of them said that he "very rarely" detects errors and the other said that she finds data errors but "not very often". Both managers suggested that data errors are more likely to occur when new products are introduced. Occasionally sales data are not aggregated correctly when new products are introduced.
"As new products are added to the database, or ... if you refresh the data because you want to sum things differently or make subtotals. That's typically when you'd see errors.Where you come into trouble is where you start subtotaling things. "
The other three consumer product managers were unable to recall an incident in which they found an error in data.
"I'm trying to remember ... I'm not sure there was a time where ... the information didn't add up or any of those kinds of things ... Actual errors where I saw it, it looked funny, I went back and checked it and you know sure enough it was miscaptured, I just never ran across that.I can't think of one off the top of my head, but it happens.
I really can't think of an instance where it's just been a wrong number. "
Three of the consumer product managers discussed strategies that they believe they use to check data for errors. One reported looking for changes in the value of key fields over time.
"I'm watching my shipments and how the output happens. And if I see some things that are inconsistent with history - cause we got some great history - when you ship twenty flavors, you've got history on every one of those, ... and if something happens, there, it seems weird. I have an ability to kind of check back, say, "Well, do we know what's going on here."
This consumer product manager added that he has never found an error in data as a result of this monitoring method.
Another consumer product manager noted that he is more vigilant about monitoring the accuracy of data immediately following the introduction of a product.
"It was a new product so I was kind of watching it more closely. I knew that there could be an error."
A third consumer product manager noted that standard checks to review the way in which data values are aggregated have been implemented by the marketing research group in her company.
"They recognize that they are not infallible, and so they've got some checks in the system. They send around - for example, whenever there's a new UPC, they will send around - they'll code it, or they'll recommend coding it a certain way, and then they'll route those sheets to all the people that work in brands. And people are supposed to look at those and say, "Okay, does this make sense, or is this in the wrong place"? And that happens sometimes. For example, there was a product that was bought out by another company. The product now is owned by another company, it's under another brand name, but the UPC didn't necessarily change. So in [the] data, it's under the wrong manufacturer."
Only one of the five consumer product managers was able to recall an incident in which a data error was missed for several months.
"I think it was some of the translation of market data. Let me think now. Yes, there was a situation, and I think it was the share base that they were using. They had incorporated some things within ... the category we were using. [The data provider] had incorporated or had some things in there that should not have been, and what happened is they were understating everybody's share. Okay, so we were there with [Beverage A] and [Beverage B] and all these other guys. And all the shares ... looked kind of low. And it was only by a couple points, they seemed a little low versus other markets we'd gone into. And they came back later - and we didn't - we kind of said, "Well, whatever, you know - people just don't drink as much of this stuff in whatever - in St. Louis, as they do in, you know, Los Angeles or whatever." So we didn't get all - because directionally they were still the same, our size versus the competition. But they came back the next period and said, 'Oops, we put in apple juice or something silly.' "
The other four consumer product managers were unable to recall an incident in they failed to detect a data error. However, three of them acknowledged that they fail to detect errors in data capture (i.e., data errors caused by an error in scanning at the point of sale).
"The errors that you will never find, and I know they happen all the time, is when a clerk doesn't want to run three flavors over the scanner. That happens all the time. I'm convinced of it...but I don't ever see it in the data, because it's so aggregated. And it wouldn't be material. I just don't believe that it would be so widespread that it would - and in only one direction. Because you do get the canceling out."
The other two consumer product managers said that they do not find data capture errors because they are not material and they cannot be corrected.
"Usually you don't find it, because stuff is so aggregated by the time we see it, that we'd never catch it. But ah - maybe at an account level - you know, if you were really looking, you might find that their sales of an apple-cinnamon flavor were a much higher proportion than what our shipments are. We usually don't see that kind of stuff, because the errors are not made on as gross of a scale, and we don't go down to that level in analysis.You assume that that's [data capture errors] just everywhere, so therefore, it's kind of like normally distributed."
As we expected, the consumer product managers perceived the payoffs for detecting data errors to be fairly low. We found that incentives for detecting errors in data were absent. One consumer product manager reported that there was a disincentive to use scanner data at the level of detail necessary to find errors. Although the consumer product managers were able to construct hypothetical scenarios in which data errors could affect their decisions, they generally agreed that data errors do not actually materially affect their decisions. The managers also agreed that suspected errors in scanning cannot be verified and corrected. However, they stated that other types of errors such as aggregation errors are not at all difficult to correct.
The managers agreed that incentives to detect errors in data do not exist in their organizations. When asked whether incentives to detect errors in data exist, four of them gave unambiguously negative responses.
"No. [laughs] Nobody has enough time to even look at the data, let alone figure out if it's right or not.You aren't encouraged to look for errors.
None. Zero.
No. "
The fifth consumer product manager responded by saying that although there are no strong incentives to detect errors in data there is an incentive to be able to explain the meaning of the data.
"I think the encouragement is in the sense that you need to understand what's going on, and you need to explain it with the data...And if you can explain it, and you know what's happening, good. You're definitely encouraged to do that. If you can't determine what's going on, then there's a disincentive to say, "Hey, I don't know what's going on." There's an incentive to say, "Hey, I don't know what's going on and I'm going to find out." ... You want to avoid a situation where you don't know what's going on. "
This manager added that he would be reluctant to publicly say that he suspected that an error existed in data until he had verified that the error actually existed.
"Before I started pointing fingers like that, I would want to be ... sure."
One of the managers noted that there is a disincentive for consumer product managers to detect data errors in his organization because consumer product managers do not want to be perceived as being as quantitative as marketing researchers.
"The culture for a lot of brand mangers is ... that the tendency is the more they use it [the data] the more they think they'll be perceived to be quantitative. There's a real irony. I think that the more I know about my business, it almost seems like I'm perceived to be very quantitative... What you've got is what people perceive to be market research data... If I need to go into this much detail [to find errors] I need market research to take care of it... I don't really think I want to look like a nerd by getting into this detail. "
The consumer product managers agreed that, for the most part, errors in scanner data do not materially affect their decisions.
"For most decisions it's very accurate.I'm not making major strategic decisions off of a one-month share.
These are directional, they're not like our profit numbers, which have to be exact.
You're wondering if decisions get made based on data that may not be accurate... I don't lose sleep over that, because my interpretation of the situation is that what we get is basically the same thing our competition gets ... And all of it is ... directional. It says, "Is your share going up? Is your share going down?" We don't look at the numbers and say, "Oh, you know, a thousand cases here or there make a difference." It's all designed to give us a pulse of what's going on...I'm getting the same thing my friends in Cincinnati and down the road are getting. And we all see it the same way...We're all in the dark on the same stuff. There are very few unique errors for [my company]."
Although the consumer product managers agreed that most data errors are not material, three of them did note that some types of errors in data could significantly affect their decisions.
They [errors in data] could be significant... For instance, if it's pricing and it looks like we're getting a certain [volume], we would then form an assumption about what price we are getting reflected at retail where in fact if that's not what's happening, for example, if our volume is really going down, we look at the pricing and it looks like it's in line with what we need, and we'd say well it's not a pricing issue. But if that was being inaccurately reflected, and that pricing was actually higher, we wouldn't know it and we wouldn't be able to respond to it.It depends on the extent of the error. It could have major ramifications...If for some reason [the data provider] was pulling the wrong volumes because you're looking at total cake mixes and they're not including seven of your [products] for some reason they put them under [another type of] cake instead of two layer cake, then all of a sudden when your volumes are dropping off drastically and you don't understand why, and you make decisions based on that. That would be an extreme example of something where you could make decisions having incorrect data.
"The only place where I can see where there's potentially a problem is when sometimes people look at promotion impact. You know, how much did I sell on promotion and it was a display or feature. There can be errors that are systematic in coding...[what should be coded as] a display, some of the volume may somehow get put in with what moved on feature."
Four of the consumer product managers agreed that investigating suspected data errors and correcting known data errors are not difficult.
"My sense is that they're [the data provider] pretty good at putting their finger on what the issue is.They have to re-run the program, or they re-run the information, and they re-submit it. And a day or two later it comes back up corrected.
All you do is call them up and say, "Can we recut this a different way?" and then you get the report the way you want it.
[The data provider is] pretty responsive... I usually get an answer within a day or so."
The types of errors that are being discussed here are errors of aggregation and classification. The managers acknowledged that it is not possible to correct data collection errors.
Table 1 summarizes the estimates of the rate of errors in data given by the consumer product managers in standard reports.
| Respondent | Serious Data Errors | Trivial Data Errors |
| 1 | 4% | 50% |
| 2 | less than 1% | interviewee doesn't know |
| 3 | 0% | 5% |
| 4 | 0-1% | 33% |
| 5 | 0% | 0% |
Each consumer product manager was asked to estimate the rate of serious and trivial data errors in the data they regularly use. Before the interviewer requested an estimate, a respondent was asked to list the types of data that he or she regularly uses. This list was used to request the base rate estimates. Thus, the types of reports (e.g., standard reports versus ad hoc reports) to which the estimates apply vary from respondent to respondent. The estimates shown in Table 1 should be interpreted as the percent of standard reports believed to contain at least one data error. For example, the first respondent shown in Table 1 estimated that four percent of the standard reports he uses contain a serious data error and fifty percent of the standard reports he uses contain a trivial data error. In one case, a respondent was unable to provide a requested estimate, as noted in Table 1.
There is considerable variance in the estimates provided by the consumer product managers. Estimates of the standard reports containing a serious data error range from zero to four percent, and estimates of the standard reports containing a trivial data error range from zero to fifty percent.
Our interviews with consumer product managers provided an interesting opportunity to examine the question of whether the perceived base rate of errors is closely related to the actual base rate of errors. Although the five interviewed consumer product managers do not all work for the same organization, they do all use scanner data provided by the two major suppliers of this data. Thus, the actual rate of data collection errors is quite similar for all of the consumer product managers. It is true that classification and aggregation errors are unique to each consumer product manager's organization. Even so, the wide disparity among base rate estimates shown in Table 1 suggests that users' perceptions of the rate of errors in data do not necessarily reflect the actual rate of errors.
Excerpts from the interview transcripts provide additional insight into the consumer product managers' perceptions about the base rate of errors in data. Some of the managers estimated that the base rate of errors is fairly low.
"I'll tell you, 99.9% of the time it's not wrong.There aren't a lot of errors...It doesn't happen that often ..I may have had one experience in four years. "
However, the consumer product managers acknowledged that the rate of error in data capture is considerably higher than this.
"I'd say generally speaking scanning errors represent about one to two percent of the volumes."You've got to realize that people are scanning stuff in a store, and even the stuff that they're scanning may be error."
These data capture errors were discounted when some of the consumer product managers were asked to estimate the error rate in the data they use. The manager who estimated the rate of serious and trivial data errors to be zero explained as followed:
" I think of them [data capture errors] as limitations. It's like my own eyesight ... you just have an understanding of the limitations of the data. So in that instance, then there are few errors."
This consumer product manager acknowledged that the data capture error rate is at least two percent, but he considers this to be a "limitation" of the data and estimates the error rate to be zero.
In this domain, the actual base rate of errors and the perceived base rate of errors appear to be quite different. Some managers appear to redefine data capture errors that cannot be verified and corrected to be limitations of the data.
Although the protocol we used to collect data in the field interviews did not specifically address error detection goals, there is evidence in several of the transcripts that the consumer product managers who were interviewed tended to assume that scanner data are accurate.
"At first you count on, or you assume the data is correct. For the most part, I don't assume the data is incorrect.Nobody has enough time to even look at the data, let alone figure out if it's right or not. I think most people are going under the assumption that it's good data.
Unless you just start seeing really weird things, you just kind of start to trust it.
Most people tend to trust the data."
As expected, not all the consumer product managers reported detecting errors. Also consistent with our a priori expectation, the perceived payoff of error detection appears to be low in this domain. If data collection errors are included in the error rate, the actual base rate of errors in data does appear to be fairly high. However, it is clear that the perceived base rate of errors varies considerably among consumer product managers and that at least some consumer product managers simply do not consider scanning errors to be data errors.
Table 2 examines the relationships between reported error detection performance and both the perceived base rate of errors and the perceived payoff of error detection within this domain.
| Respondent | Detected an Error | Failed to Detect an Error | Base Rate of Errors | Payoffs of Error Detection |
1 |
yes |
yes |
1 |
Inc = low
Mat = low V/C = moderate |
2 |
yes |
no |
3 |
Inc = low
Mat = low V/C = high |
3 |
no |
no |
3 |
Inc = low
Mat = low V/C = moderate |
4 |
no |
no |
2 |
Inc = low
Mat = moderate V/C = high |
5 |
no |
no |
4 |
Inc = low
Mat = low V/C = low |
Inc = Incentives to detect errors
Mat = Materiality of errors
V/C = Ease of verifying and correcting errors
The second and third columns of Table 2 indicate whether each respondent reported a specific error detection incident and a specific incident in which a data error was missed. In the fourth column, respondents are ranked from highest to lowest on the basis of their estimates of the base rate of serious and trivial errors in data (see Table 1 for the estimates used to rank the interviewees). Table 2 shows that Respondent #1 provided the highest base rate estimate and Respondent #5 provided the lowest base rate estimate. Notice that both Respondent #2 and Respondent #3 are ranked third. This means that it was not possible to reliably determine which interviewee provided the higher estimate. In this case, Respondent #2 gave an estimate of less than one percent for serious errors and said she could not provide an estimate for trivial errors while Respondent #3 gave an estimate of between zero and one percent for serious errors and gave an estimate of five percent for trivial errors. The fifth column of this table summarizes each interviewee's perceptions about incentives, the materiality of data errors, and the ease of verifying and correcting data errors. Respondents were assigned a judgment of High, Moderate, or Low for each of these three categories. For example, Respondent #1 believes that incentives to detect data errors are low, that the materiality of data errors is low, and that it is moderately difficult to verify and correct suspected data errors.
We use the information presented in Table 2 to draw several tentative conclusions. First, there does not appear to be a strong relationship between error detection performance and perceived payoffs of error detection within the domain of consumer product managers. The profiles in the fourth column do not differ substantially for those respondents who reported detecting data errors and those who did not. Second, there may be a weak link between error detection performance and estimates of the base rate of errors within the domain. Notice that Respondent #1, who reported both an incident when he detected an error and an incident in which he learned that an error had been missed, is the respondent with the highest base rate estimate. As discussed earlier, the actual rate of errors is fairly constant in this domain because all of the consumer product managers use scanner data acquired from one of two national suppliers.
We believe that our study of consumer product managers has contributed to the research about how professionals who use data evaluate data quality. In an earlier study, conducted as part of this research stream, we interviewed ten actuaries. All of them reported that they had detected errors in data. Our consumer product manager interviews indicate another professional domain where some of the people interviewed reported that they detected data errors. This suggests that deficiencies in data error detection are not a rare organizational occurrence. We found that three of the five consumer product managers we interviewed were unable to recall an incident in which they found an error in data. This supports our belief that some data users who are unable to describe incidents in which they have detected data errors have really detected data errors. Our research also suggests that successful error detection strategies may be similar across professions.
There are three general approaches to the improvement of organizational effectiveness in response to data quality problems. These approaches are depicted in Figure 2 and described below.
| Input ------------> | Stored Data------------> | Use |
| Prevention, Detection, and Correction |
Detection and Correction |
Detection and Modification of Use |
Data Input Procedures. Our first approach focuses on the prevention, detection, and correction of inaccurate and incomplete data as they are input to organizational databases.
Stored Data Procedures. Our second approach focuses on the detection and correction of inaccurate data that are stored in organizational databases.
Detection of Errors by Users. Our third approach focuses on interventions that might improve human ability or motivation when detecting errors in organizational database reports.
Concomitantly, our research shows that there are three ways that data errors can occur in the process of creating and disseminating product data for consumer product managers.
First, errors may occur when items are scanned incorrectly in retail stores. For example, consider a transaction in which a customer purchases three items, each a different flavor of the same product. A cashier may incorrectly record this purchase by scanning one of the items three times and failing to scan the other two items. The consequence of this error is that the quantity sold data for all three flavors of the product is incorrect. Omissions in scanning may also occur. For example, cashiers may fail to scan a heavy item like a fifty pound bag of dog food if the customer has placed the bag under the basket of a grocery cart. Cashiers may also fail to scan items if bar codes are not clear or if scanning equipment is not operating properly. Rather than keying in the bar code data, a cashier may choose to simply enter the price of the product.
Next, data errors may occur when pricing information is entered in a store's pricing database. Although the accuracy of pricing data has improved over time, there is evidence that scanners still give the wrong price for items all too often. A study recently conducted by the office of the Michigan Attorney General found an average error rate of thirteen percent for sale items. The highest error rate found for sale items was 23.7 percent, and the lowest error rate was 5.5 percent. An error rate of three percent was found for non-sale items (Store scanners', 1997). The scanning errors found in this study occur when databases storing current pricing data are not properly maintained. This is a critical problem for consumer product managers because they use scanner data to evaluate the effectiveness of promotions and sales. Without accurate pricing data, it is impossible to understand to impact of promotional programs.
Finally, errors may occur when data are aggregated and categorized by the organizations that sell this data to consumer product companies. For example, sales data may be incorrectly categorized by product line or by sales territory.
Three of the consumer product managers interviewed for this study said that while they are certain that scanning and pricing errors exist, they do not attempt to find this type of error. Instead, their error detection efforts are focused on finding errors of aggregation and categorization. Given their experience, we carefully considered possible methods to improve data quality by improving data input procedures. While our research has primarily addressed data error correction by end users, two of our findings may be applicable to data input procedures. These are explicit task goals in error detection and incentives for improving accuracy.
Our research suggests that explicit task goals and clear directions play an important role in data error detection (Klein et al., 1997). It appears that some cashiers do not have a clear understanding of the critical importance of accurate sales data and do not believe that creating an error free record of every transaction is an important part of their job. While cashiers may be motivated to charge each customer the correct amount for each purchase, they may be less motivated to ensure that each individual item purchased is actually picked up and scanned. Training that emphasizes the importance of recording the sale of each individual item may improve the accuracy and completeness of sales data at the collection point.
We found error detection performance is strongly affected by incentives (Klein et al., 1997). Given the modest compensation of cashiers, providing cash incentives to reward scanning accuracy may greatly improve the accuracy and completeness of sales data. One possible approach is to randomly sample the accuracy of individual cashiers and provide cash bonuses that are a function of scanning accuracy. Incentives may also improve the quality of data maintained in stores' pricing databases. Organizations that purchase scanner data from retail stores could design an incentive scheme where stores found to maintain accurate pricing databases would receive a higher price for their scanner data than stores found to have less accurate pricing databases. Stores could be randomly sampled to determine whether sales and promotions are accurately recorded in pricing databases.
These suggestions for improving the accuracy of data during data collection may provide an avenue for future research. We have found little information about how to better motivate, compensate, and train low-level employees involved in data collection. Future studies could examine the efficacy of programs designed to improve data quality at the source through incentives and training about appropriate goals. We believe that future studies should also focus on determining the appropriate characteristics for incentive schemes and training programs, since it is not yet known what types of rewards would have the greatest impact on cashiers' scanning techniques, or on the efforts of store managers to ensure that pricing databases contain accurate, up-to-date information.
We believe that significant changes in retail store management may be necessary to effectively implement programs designed to improve the motivation, compensation, and training of employees who collect data. We believe that a cultural shift won't occur until the end users of the data (e.g., consumer product managers) demand more accurate and complete data and support this demand with a willingness to pay a higher price for higher quality data.
While our research focused on the task of collecting scanner data used by consumer product managers, future research should address data and information quality in other consumer-based organizations.
We believe that future research should apply quality assurance methods developed by American manufacturing to prevent errors "up front.". Over a decade ago, American manufacturing quality shifted from an "inspect/repair" methodology to an emphasis on preventing defects (Duncan, 1994). Manufacturing quality tools like statistical process control and failsafing were developed to prevent defects in manufacturing. While our research focused primarily on data error detection, researchers and practitioners interested in data quality assurance must also focus on preventing defective data. A shift in emphasis from the detection of data errors to the prevention of data errors could greatly improve information systems productivity and improve information systems efficacy. We believe that future research should focus on the application of tools used in manufacturing quality - statistical process control, failsafing, and concurrent engineering - to the processes associated with the collection, storage, and output of data in information systems.
Agmon, N., and N. Ahituv. 1987. Assessing data reliability in an information system. Journal of Management Information Systems 4 (2): 34-44.
Ballou, D. P., and H. L. Pazer. 1985. Modeling data and process quality in multi-input, multi-output information systems. Management Science 31: 150-162.
Ballou, D. P., and H. L. Pazer. 1987. Cost/quality tradeoffs for control procedures in information systems. OMEGA: International Journal of Management Science 15: 509-521.
Ballou, D. P., and H. L. Pazer. 1995. Designing information systems to optimize the accuracy-timeliness tradeoff. Information Systems Research 6: 51-72.
Ballou, D. P., H. L. Pazer, S. Belardo, and B. Klein. 1987. Implications of data quality for spreadsheet analysis. Data Base 18 (3): 13-19.
Ballou, D. P., and G. K. Tayi. 1989. Methodology for allocating resources for data quality enhancement. Communications of the ACM 32: 320-329.
Bansal, A., R. J. Kauffman, and R. R. Weitz. 1993. Comparing the modeling performance of regression and neural networks as data quality varies: A business value approach. Journal of Management Information Systems 10: 11-32.
Bowen, P. L. 1992. Managing data quality in accounting information systems: A stochastic clearing system approach. Abstract in Dissertation Abstracts International 54-03A (1992). Accession number 9319179.
Campbell, J. P. 1990. Modeling the performance prediction problem in industrial and organizational psychology. In Handbook of Industrial and Organizational Psychology. 2d ed. edited by M. D. Dunnette and L. M. Hough, Vol. 1, 687-732. Palo Alto, Calif.: Consulting Psychologists Press.
Campbell, J. P., and R. D. Pritchard. 1976. Motivation theory in industrial and organizational psychology. In Handbook of Industrial and Organizational Psychology, edited by M. D. Dunnette, 63-130. Chicago: Rand McNally College Publishing Company.
Consumer enemy no. 1. 1991. Newsweek 28 October, 42, 47.
Corman, L. S. 1988. Data integrity and security of the corporate data base: The dilemma of end user computing. Data Base 19: 1-5.
Cryer, E. H., J. R. Bettman, and J. W. Payne. 1990. The impact of accuracy and effort feedback and goals on adaptive decision behavior. Journal of Behavioral Decision Making 3: 1-16.
Davis, G. B., J. Neter, and R. R. Palmer. 1967. An experimental study of audit confirmation. Journal of Accountancy 123 (6): 36-44.
Davis, G. B., and M. H. Olson. 1985. Management information systems: Conceptual foundations, structure, and development. New York: McGraw-Hill Book Company.
Dead farmer syndrome haunts efforts to trim USDA offices. 1992. Minneapolis Star Tribune19 April, p. 5A.
Duncan, W. L. 1994. Manufacturing 2000. New York: American Management Association.
Ericsson, K. A., and W. G. Chase, W. G. 1982. Exceptional memory. American Scientist 70: 607-614.
Flanagan, J. C. 1954. The critical incident technique. Psychological Bulletin 51: 327-358.
Fox, C., Levitin, A., and T. C. Redman. 1993. The notion of data and its quality dimensions. Information Processing & Management 30: 9-19.
Gaba, A., and R. L. Winkler. 1992. Implications of errors in survey data: A Bayesian model. Management Science 38: 913-925.
Garfinkel, R. S., A. S. Kunnathur, and G. E. Liepins. 1986 . Optimal imputation of erroneous data: Categorical data, general edits. Operations Research 34: 744-751.
Ham, J., D. Losell, D., and W. Smieliauskas. 1985. An empirical study of error characteristics in accounting populations. The Accounting Review 60: 387-406.
Huh, Y. U., F. R. Keller, T. C. Redman, and A. R. Watkins. 1990. Data quality. Information and Software Technology 32: 559-565.
Janson, M. 1988. Data quality: The Achilles heel of end-user computing. OMEGA: International Journal of Management Science 16: 491-502.
Johnson, E. J., and J. W. Payne. 1985. Effort and accuracy in choice. Management Science 31: 395-414.
Johnson, J. R., R. A. Leitch, and J. Neter. 1981. Characteristics of errors in account receivable and inventory audits. The Accounting Review 56: 270-293.
Johnson, P. E., S. Grazioli, and K. Jamal. 1992a. Fraud detection: Sources of error in a low base-rate world. Paper presented at 10th European Conference on Artificial Intelligence Research Workshop on Expert Judgment, Human Error, and Intelligent Systems, Vienna, Austria.
Johnson, P. E., S. Grazioli, K. Jamal, and I. A. Zualkernan. 1992b. Success and failure in expert reasoning. Organizational Behavior and Human Decision Processes 53: 173-203.
King, N. 1994. The qualitative research interview. In Qualitative methods in organizational research, edited by C. Cassell and G. Symon, 14-36. Thousand Oaks, Calif.: Sage Publications.
Klein, B. D. 1997. How do actuaries use data containing errors? Models of error detection and error correction. Information Resources Management Journal 10 (4): 27-36.
Klein, B. D., D. L. Goodhue, and G. B. Davis. 1996. Conditions for the detection of data errors in organizational settings: Preliminary results from a field study. In Proceedings of the 1996 conference on information quality. Cambridge, MA: MIT.
Klein, B. D., D. L. Goodhue, and G. B. Davis. 1997. Can humans detect errors in data? Impact of base rates, incentives, and goals. MIS Quarterly 21 (2): 169-194.
Knight, B. 1992. The data pollution problem. Computerworld 26 (39): 81-83.
Laudon, K. C. 1986. Data quality and due process in large interorganizational record systems. Communications of the ACM 29: 4-11.
Madnick, S. E., and R. Y. Wang. 1992. Introduction to the TDQM research program. Total Data Quality Management Research Program Working Paper, no.92-01.
Mason, R. O. 1986. Four ethical issues of the information age. MIS Quarterly, 10: 5-12.
Miles, M. B., and A. M. Huberman. 1994. Qualitative data analysis. Thousand Oaks, Calif.: Sage Publications.
Morey, R. C. 1982. Estimating and improving the quality of information in a MIS. Communications of the ACM 25: 337-342.
Naus, J. I. 1975. Data quality control and editing. New York: Marcel Dekker.
O'Leary, D. E. 1993. The impact of data accuracy on system learning. Journal of Management Information Systems 9: 83-98.
O'Neill, E. T., and D. Vizine-Goetz. 1988. Quality control in online databases. In Annual review of information science and technology, edited by M. E. Williams, 125-156. Amsterdam: Elsevier Science Publishers.
Paradice, D. B., and W. L. Fuerst. 1991. An MIS data quality methodology based on optimal error detection. Journal of Information Systems 5 (1): 48-66.
Parsaye, K., and M. Chignell. 1993. Data quality control with SMART databases. AI Expert 8 (5): 23-27.
Payne, J. W. 1982. Contingent decision behavior. Psychological Bulletin 92: 382-402.
Payne, J. W., J. R. Bettman, and E. J. Johnson. 1988. Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition 14: 534-552.
Percy, T. 1986. My data, right or wrong. Datamation 32 (11): 123-128.
Redman, T. C. 1992. Data quality: Management and technology. New York: Bantam Books.
Redman, T. C. 1995. Improve data quality for competitive advantage. Sloan Management Review 36 (2): 99-107.
Ricketts, J. A. 1990. Powers-of-ten information biases. MIS Quarterly 14: 63-77.
Stone, M., and B. Bublitz. 1984. An analysis of the reliability of the FASB data bank of changing price and pension information. The Accounting Review 59: 469-473.
Store scanners' accuracy improves. 1977. Detroit Free Press, 27 November, p. 2E.
Svanks, M. I. 1988. Integrity analysis. Information and Software Technology 30: 595-605.
Wand, Y., and R. Y. Wang. 1994. Anchoring data quality dimensions in ontological foundations. Total Data Quality Management Research Program Working Paper, no. 94-03.
Weber, E. U., U. Bockenholt, D. J. Hilton, and B.Wallace. 1993. Determinants of diagnostic hypothesis generation: Effects of information, base rates, and experience. Journal of Experimental Psychology: Learning, Memory, and Cognition 19: 1151-1164.
Zmud, R. W. 1978. An empirical investigation of the dimensionality of the concept of information. Decision Sciences 9: 187-195.