Volume 3 Number 1 Copyright 1997
Survey Measurement and Process Quality
Lars Lyberg et al. (editors)
New York: Wiley-Interscience, 1997
ISBN 0-471-16559-X, 808 pages, $94.95
Reviewed by F. DeWitt Kay, Jr.
In April 1995, the Survey Research Methods Section of the American Statistical Association (ASA) convened the International Conference on Survey Measurement and Process Quality, held in Bristol, United Kingdom. The 34 invited papers from this conference have been revised and collected in Survey Measurement and Process Quality (hereafter, SMPQ). Of the additional 113 contributed papers, 60 were published separately by the ASA under the title, SMPQ: Proceedings of the International Conference on Survey Measurement and Process Quality ($40). For additional information about the proceedings, readers may wish to contact ASA directly (asainfo@amstat.org or call 703-684-1221).
SMPQ authoritatively details recent linkages forged between total quality management (continuous quality improvement) techniques and large-scale (e.g., national) survey efforts. While focusing on routinized survey operations, whose repetitive nature offers ample opportunity for ongoing process evaluation and informed process modification, its breadth of topics suggests that even those designing the occasional survey will find much useful guidance here. Like the conference itself, SMPQ is truly international in its coverage of best practices widely employed in North America, Europe, and Australia for promoting survey quality.
Five major sections follow O'Muircheartaigh's introduction and comprehensive review of earlier efforts at improving survey accuracy and reducing nonsampling error ("Measurement Error in Surveys: A Historical Perspective"). Each section contains seven chapters, except for Section C (five chapters). Section and chapter organization roughly parallels the sequence of operations in a typical complex survey. Overall, the constituent chapters represent a timely guide to enhancing survey accuracy and usefulness through better design of the entire measurement process -- from survey conceptualization through interpretation of statistical findings.
Section A: Questionnaire Design
Schwarz opens the first section with a review of issues attending survey-instrument development, particularly cognitive aspects of the response process ("Questionnaire Design: The Rocky Road from Concepts to Answers"). In "From Theoretical Concept to Survey Question," Hox reminds researchers that clearly defined constructs and scientific concepts must drive the design of meaningful survey questions. In "Why Are There So Few Formal Measuring Instruments in Social and Political Research?," Heath and Martin cite the inadequate attention given to psychometric properties of survey instruments. Johnson et al. discuss cross-cultural considerations when surveying diverse populations. A chapter by Wanke and Schwarz explores the use of buffer items in reducing question-order effects.
Krosnick and Fabrigar provide a useful update of studies that examine rating-scale construction, particularly the optimum number of scale points, their labeling, and the effects of including no-opinion filters. "Towards a Theory of Self-Administered Questionnaire Design" (Jenkins and Dillman) enumerates desirable graphic-design properties for respondent-completed survey instruments. Invoking visual-perception principles and cognitive science, the authors demonstrate how proper layout reduces respondent error.
Section B: Data Collection
Beginning with an overview of data-collection methods and their impacts on survey quality (de Leeuw and Collins), the second section contains several chapters that examine recent approaches to response capture. Blythe reports on the status of automatic speech recognition systems, while Couper et al. evaluate computer assisted personal interviewing (CAPI). Nicholls et al. discuss how these newer technologies have affected data quality, in general their findings are encouraging. Dykema et al. use interaction coding of standardized health interview field responses to determine the impact of interviewer-respondent interactions on information quality. Jobe et al. examine the effects of interview mode on eliciting answers to sensitive questions. Lastly, Scott examines data quality issues that arise when survey respondents are children.
Section C: Post Survey Processing and Operations
Lyberg and Kasprzyk review approaches to data editing, coding, and capture, examining their manifold influences on data quality. Bethlehem advocates using integrated control systems for survey-processing operations and organizational redesign to reduce processing errors. The chapter includes a discussion of meta-data concepts. Conrad reports on the increased use of expert systems for survey edits and explores the U.S. Bureau of Labor Statistics' COMPASS and MatchMaker systems. Grandquist and Kovar's "Editing of Survey Data: How Much is Enough?" reviews editing-cost studies and concludes that extensive editing yields sharply diminishing data-quality value. The authors advocate selective editing and greater focus on error-prevention rather than error-correction. The section concludes with a chapter by Campanelli et al. that examines the quality of occupational coding in the United Kingdom.
Section D: Quality Assessment and Control
Total Quality Management and Continuous Quality Improvement practitioners should find much to like in this section. Cathryn Dippo (U.S. Bureau of Labor Statistics) contributes a short, informative chapter, "Survey Measurement and Process Improvement: Concepts and Integration." Morganstein and Marker provide an overview of useful CQI practices in "Continuous Quality Improvement in Statistical Agencies." Their chapter catalogs promising strategies for the survey industry and gives a clear explanation of current best method (CBM) development. Colledge and March's chapter summarizes data-quality practices at 16 national statistical agencies.
Gross and Linacre's chapter highlights comparability problems encountered when deriving consistent estimates from cross-business surveys. Esposito and Rothgeb's chapter, "Evaluating Survey Data: Making the Transition from Pretesting to Quality Assessment," summarizes redesign of the U.S. Census Bureau's Current Population Survey to illustrate how multi-method evaluation might produce improved survey questionnaires. The chapter begins with a review of evaluative techniques and concludes with thoughtful prescriptions for an idealized quality assessment program. Batcher and Scheuren chronicle the design and conduct of a computer assisted telephone interview (CATI) quality study of IRS employees who provide telephone tax assistance. Hapuarachchi et al. note that ongoing data quality monitoring often involves autocorrelated measurements. They recommend and illustrate statistical techniques that accommodate such serial dependencies.
Section E: Error Effects on Estimation, Analysis, and Interpretation
The final section is intended for a sophisticated statistical audience. Non-specialists familiar with log-linear techniques might profitably tackle opening chapters: (#27, Biemer and Trewin) "A Review of Measurement Error Effects on the Analysis of Survey Data"; (#28, Kuha and Skinner) "Categorical Data Analysis and Misclassification"; and (#29, van de Pol and Langeheine) "Separating Change and Measurement Error in Panel Surveys With an Application to Labor Market Data." The book's last four chapters present sophisticated estimation procedures developed for specific applications.
The seven co-editors and Wiley's editorial team have produced a work refreshingly free of errors and misprints. Unfortunately, SMPQ lacks a comprehensive author index. One must search instead through reference lists at the end of each chapter; however, these are worth consulting as useful guides to recent literature.
Any organization that uses survey methodology, even occasionally, will surely want a copy of Survey Measurement and Process Quality in its library. Given the book's high cost ($94.75), individuals considering purchasing the volume might want first to confirm its applicability to their specific interests. The book is occasionally available at discount from the American Statistical Association.
Contrary to the publisher's promotional claims on the rear cover, the reviewer has trouble envisioning SMPQ (other than Section A) as an appropriate supplemental text for undergraduate and graduate courses, notwithstanding its obvious relevance in agency and business settings.
Go to:
Comments: dqemail@aol.com (10/04/98)