Volume 3 Number 1 Copyright 1997
Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis:
Methods for Quantitative Synthesis in Medicine
Diana B. Petitti
New York: Oxford University Press, 1994
ISBN: 0-19-507334-7, 256 pages, $49.50
Reviewed by Betty C. Jung
Key Words: Health Care, Cost-Benefit Analysis, Quality
How do physicians determine what's best for their patients? How do patients decide what's best for themselves? Due to rapid changes in health care delivery within the past 5-10 years, the doctor-patient relationship must answer to third-party payers (e.g., HMOs) who insist that they know what excellent medical care is. Consequently, medical professionals are spending much time and effort researching fields like evidence-based medicine, practice guidelines, and clinical prediction rules. [1] While these guidelines are designed to provide direction for clinical decision-making, the critics of both "managed care" and the legal complexities managed care spawns are numerous and vocal. [2,3]
Moreover, concomitant with such organizational upheavals are the changing dimensions of the disease process. For example, at the beginning of this century many people died quickly from infectious diseases. At century's end -- battling emerging infectious agents unbowed by antibiotics -- we face the growing need to manage those who were acutely infected and will be chronically ill for years with diseases like AIDS, hepatitis, and tuberculosis. And the population of elderly Americans with multiple chronic diseases is growing.
The cost of treatment often no longer justifies the results. Often, a cure is no longer a viable resolution to the disease process, or even a treatment. Being well is sometimes no longer an absolute or achievable state of being, but a transient state between being ill and being better. When we can no longer reclaim our previous state of being before we became ill, we are faced with the need to rethink what it means to be well.
This is never easy. We justify the cost of good health by contemplating the quality of our lives. This has become a "quality of health" issue because we must deal with the health care system more than we really want to. Health care, which was never affordable for the seriously ill, is now unbearably expensive because economies of scale don't apply to such cases.
Physicians want the freedom to do what needs to be done. Third-party payers second-guess physicians by refusing to pay for everything physicians may want to do because they judge some physicians' actions are not medically necessary. Recently, physicians and third-party payers have rushed to justify their positions by fleeing to the world of statistics. Evidence quantified is action justified.
Unfortunately, this trend has not been good for patients, who were formerly clients, but are now considered consumers. Suddenly we have become an audience in need of marketing, falling victim to the constant bombardment of report cards, rankings, and ratings, as well as being privy to the seemingly contradictory analyses of health data, all of which are "significant" in some way. Of those who have access to such information, only about a third use it to make health care decisions. Specifically, 34% use the information to choose a health plan, 35% to choose a physician, and 30% to choose a hospital. [4] It should be noted that it was unclear whether the percentages reported were overlapping or mutually exclusive.
And when people need health services, it is not unusual for patients to be told of all the possible adverse effects of a procedure or treatment they must undergo, or to be asked to decide what to do next on the basis of the probability of good and bad outcomes for each option. It is hard to say how much of this "information" is genuinely understood, when most people do not understand the concept of probability (think of lottery tickets and the thriving economies of Las Vegas and Atlantic City). Nevertheless, it appears that we will be increasingly forced to deal with statistics-laden health information in the future.
Diane Petitti's Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis: Methods for Quantitative Synthesis in Medicine [5] is an excellent basic text for understanding the proper use of the statistical methods being used, abused, and misused by those performing analyses of secondary health data. Results of such analyses are often used for developing health policy. Organized in 15 chapters, with an outline format, Dr. Petitti's book attempts to describe the design, conduct, analysis, and interpretation of synthetic studies. Though written for physicians, the text can be understood by college-level readers interested in learning about these methodologies.
The first chapter provides an historical overview of meta-analysis and cost-effectiveness analysis studies, as well as a context within which these studies can be linked in terms of addressing a particular health problem. The second chapter provides a brief overview of methodology, with examples of how to conduct each analysis. Chapter 3 is about planning such studies, and Chapter 4 describes methods of retrieving information for these studies; from accessing computerized databases (e.g., Medline), to considering fugitive literature (government reports, book chapters, conference proceedings, published dissertations), to accounting for publication bias. Chapter 5 covers the issues of reliability and validity in data collection.
"Advanced Issues in Meta-Analysis" (Chapter 6) expands on the second chapter's coverage of developing eligibility criteria and a study design for meta-analytic studies. Issues such as the effects of using multiple publications from the same study population, sample size and length of follow-up restrictions, differences in treatments and outcomes, incomplete information, time horizon, and estimates of effect are also covered. The author devotes a subsection to assessing study quality using rating scales. "Statistical Methods in Meta-Analysis" (Chapter 7) explains the difference between fixed-effects and random-effects statistical methods. The author includes descriptions of fixed-effects methods (Mantel-Haenszel, Peto, general variance-based, confidence interval) and random-effects methods (DerSimonian and Laird), along with statistical tests for homogeneity (with examples).
Chapter 8, "Other Statistical Issues in Meta-Analysis," deals with continuous scale measurements, estimating trends, modeling, vote counting, and statistical appproaches to publication bias. "Complex Decision Problems" (Chapter 9) explains how to do decision analyses when there is more than one outcome, more than two alternative treatments or interventions, and many intervening events. Other topics include estimating life expectancy and using Markov models when there are transitions in and out of various states of health. Chapter 10 discusses probability estimation based on published sources, meta analyses, expert panels, and personal experience. Methods to account for uncertainty in probability estimates (e.g., Bayesian statistics) are covered, and the author presents a critique of these methods.
Chapter 11 is about utility analysis, which is a decision analysis that incorporates measures of preference. This chapter covers measuring preference for health states, measurement scale development, and incorporating these preference measures into decision analysis. "Advanced Cost-Effectiveness Analysis" (Chapter 12) concentrates on key concepts like estimating costs and discounting costs and inflation. Sensitivity analyses are covered in Chapter 13. Chapter 14 outlines how one reports these synthetic studies, and Chapter 15 delineates the limitations associated with each of these methods.
Meta-analysis is the basic synthetic study from which decision analyses, and subsequent cost-effectivness analyses should emerge. This does not always happen. Many times these studies are conducted independently of one another, to the detriment of all. The main purpose of meta-analysis is to combine the results of many studies. Meta-analysis studies are also known as systematic reviews and are gaining medical community acceptance, [6] although there is a question of what happens when multiple reviews yield different clinical recommendations. [7]
To do meta-analysis studies well is a challenge. Biomedical research is conducted worldwide, and not all research is reported in English. In one particular meta-analytic study, 20% of relevant studies were in languages other than English. The Cochrane Collaboration, an international network, is being developed to facilitate the unbiased collection of data from worldwide clinical trials. [8] Moreover, the quality of data retrieved by computer searches can vary greatly. [9] Even a well-trained Medline searcher may miss half of all available studies. [10] Finally, it should be noted that meta-analytic studies are inappropriate if there are large differences between trials in the choice of patients, interventions, and measurements of effects. [11]
According to recent literature, life expectancy as an outcome measure of decision analysis and cost-effectiveness analysis is associated with issues of ethics, values, and morals. When used as a measure of effectiveness, an intervention that prolongs life will have the smallest effect on gain in life expectancy in the group with the shortest life expectancy. Furthermore, the cost per year of life gained will be greatest in the group with the shortest life expectancy, in the absence of intervention. [12] In essence, those with the shortest life expectancy usually lose out in decision and cost-effectiveness analyses.
The main purpose for conducting decision analysis studies is to determine which treatment alternative is the best. Decision analysis is a method likely to be questioned because many of its assumptions and probability estimates are derived. Petitti notes that very little critical evaluaton of this method has been done. How choices are formulated for analyzing individual preferences is influenced by a "framing" effect. [13] While they offer much leeway in predicting outcomes for what-if scenarios, decision analyses could gain more credibility if their probability estimates were based on well-done meta-analyses.
The main purpose for conducting cost-effectiveness analysis (CEA) studies is to determine which treatment option is the most cost-effective. Most important in CEA studies is the need to specify the perspective used, since that will determine which costs are included and which economic outcomes are considered benefits. [14] Many CEA studies have been poorly done. A federal panel recently cited many deficiencies. Among those mentioned were: (1) not defining the perspective of the analysis; (2) effectiveness data were inadequate or difficult to evaluate; (3) cost data were inadequate or could not be generalized; (4) choice of a comparison intervention distorted the cost-effectiveness ratio; (5) inadequate representation of the effect of time on the future value of health and dollars; and (6) inadequate representation of uncertainty regarding key assumptions. These deficiencies limit the analyst's ability to compare different CEA studies. [15] The abuse of cost-effectiveness studies has resulted in the call for evaluating the role of medical necessity and cost-effectiveness in medical decision-making. [16] But suggestions like adding laboratory data to administrative data [17] show improvement is possible.
A basic problem with CEAs is the use of charge data as a proxy for cost data. [18] This is based on the assumption that analyzing costs will improve the delivery of health care. Costs, however, have very little to do with the quality of health care one receives, much less what is charged for health care services. For example, what one hospital or provider charges for a particular procedure may not necessarily be what another charges. Geographic variations exist. More specifically, how much hospitals or providers charge for a particular procedure is no reflection of the quality of the procedure performed. There is no guarantee that if a patient pays more, the procedure will be done better, nor does paying less mean the procedure was not done correctly. In addition, what third-party payers are willing to pay providers may only be a fraction of what they charge. So one can see why there is a critical need to find better ways to measure cost before CEAs can be used by health policy-makers for making health care decisions.
Surprisingly, quality of care is not a major determinant in a patient's decision to initiate a malpractice claim. [19] This would suggest that bad outcomes are not necessarily viewed the same way by physicians, third-party payers, and patients. What, then, makes for quality of care? Can it be quantified?
Within the purview of data quality there are some caveats to consider. First, much of the data used in synthetic studies are secondary data, which are generally not as good as the primary data we set out to collect in consistent ways for studies we have designed. To use secondary data requires researchers to assume that the quality of the studies and the data that were collected during the studies are excellent. It is usually difficult or impossible to determine if the data are of high quality when they were collected for another purpose. Nor is it known whether or not such data would be truly appropriate for the current synthetic study without some evaluation.
Even when meta-analysts use a rating system to evaluate a study's quality, the exercise is subjective because "quality" can be defined differently by different researchers. However, experts in the field, like Chalmers and Altman, place greater emphasis on study design (60%) than on statistical analysis (30%) and data presentation (10%). [20] Insofar as bias is concerned, Light and Pillemer note that any systematic bias found in a dataset is not eliminated by analyzing a subset of such a dataset. [21] Nor does performing meta-analysis using a group of non-experimental studies remove the possibility of bias and uncontrolled confounding that may exist in the original studies. [22] Statistical methodology cannot compensate for poor data quality!
Second, publication bias remains a thorny issue, not only in medical research, but in other scientific disciplines as well. It cannot be said that unpublished studies are necessarily poorly designed studies, though some may well be. The reality of academic life, that being able to show a statistically significant difference is a major determinant of a study's "publishability," is an injustice to well-designed studies that show nothing statistically significant but may be just as scientifically valid as those that do. In medical research, statistical significance does not necessarily mean clinical significance. Petitti does not recommend using statistical approaches to compensate for publication bias. [23]
Third, there is a need in the medical literature for the structured abstract that concisely summarizes the study so it can be easily understood at a glance and meaningfully integrated into relevant meta-analytical studies. For example, the AMA has published uniform guidelines for reporting biomedical studies. These also include instructions for preparing structured abstracts. [24]
Researchers should realize that not everyone is interested in reading lengthy studies. The effort expended in developing a decent structured abstract of one's study is repaid many times over by its being selected for meta-analysis studies and thoughtfully considered by researchers accessing online databases. Given the shrinking pool of research grants, consolidating the work of many researchers studying the same problem could tone up a body of knowledge that is becoming too flabby to meet future needs.
In summary, Petitti is an excellent teacher. Her generous use of relevant examples helps the reader comprehend methodological concepts, adapted from other disciplines, that need to be understood in order to gain the most from current research findings, even though they may seem irrelevant at times to the practice of medicine. For those involved with health policy, this textbook is a godsend. Her book is worth owning for its excellent coverage of analytical methods that are being used with increasing frequency in health services research. When these methods are used properly, statistics is a useful tool in helping medicine handle the growing uncertainty that comes with better technology. Whether or not the quality of health care significantly influences the quality of life for those who want to feel better remains to be seen.
References
1. A. Laupacis, N. Sekar, and I.G. Stiell, "Clinical Prediction Rules. A Review and Suggested Modifications of Methodological Standards," JAMA, Vol. 277, no. 6, (February 12, 1997): 488-494.
2. C.M. Nunn, "Pathways, Guidelines, and Cookbook Medicine: Are We All Becoming Betty Crocker?," JCOM, Vol. 4, no. 1, (January-February 1997): 17-24.
-- R.I. Horwitz, "The Dark Side of Evidence-Based Medicine," Cleveland Clinic Journal of Medicine, Vol. 63, no. 6, (October 1996): 320-323.
-- B. Jancin, "'Junk Science' Practice Guidelines Don't make the Grade," Internal Medicine News, 15 March 1995, 36.
-- S. Boschert, "Many Practice Guidelines Produced by Insurers and Specialty Societies Called Self-Serving," Internal Medicine News and Cardiology News, 15 August 1994, 30.
3. F. Kostreski, "PruCare Sued for Using Length-of-Stay Guidelines," Internal Medicine News, 15 February 1997, 57.
4. J.R. Rose, "Quality of Care. People Say It Matters - But Does It Really?", Medical Economics, Vol. 74, no. 2 (January 27, 1997): 26-32.
5. D.B. Petitti, Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis: Methods for Quantitative Synthesis in Medicine, (New York, Oxford University Press, 1994).
6. C.D. Mulrow, D.J. Cook, and F. Davidoff, "Systematic Reviews: Critical Links in the Great Chain of Evidence," Annals of Internal Medicine, Vol. 125, no. 5 (March 1, 1997): 376-380.
7. D.J. Cook, C.D. Mulrow, and R.B. Haynes, "Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions," Annals of Internal Medicine, Vol. 125, no. 5 (March 1, 1997): 376-380.
8. I. Chalmers and D.G. Altman, editors, Systematic Reviews, (London, BMJ Publishing Group, 1995).
9. Petitti, 52-56.
10. Chalmers and Altman, 27.
11. Chalmers and Altman, 11.
12. Petitti, 225.
13. Petitti, 219-220.
14. Petitti, 32.
15. The Panel on Cost-Effectiveness in Health and Medicine, Cost Effectiveness in Health and Medicine / Project Summary, (Washington, D.C., U.S. Public Health Service - Office of Public Health and Science, 1996): 5.
16. P.A. Glassman, et. al., "The Role of Medical Necessity and Cost-Effectiveness in Making Medical Decisions," Annals of Internal Medicine, Vol. 126, no. 5 (January 15, 1997): 152-156.
17. M. Pine, et al., "Predictions of Hospital Mortality Rates: A Comparison of Data Sources," Annals of Internal Medicine, Vol. 126., no. 5 (March 1, 1997): 347-354.
18. Petitti, 220-222.
19. "Malpractice Suit Risk Tied to Lack of Communication," American Medical News, Vol. 40, no. 8 (February 24, 1997): 20.
20. Petitti, 85-86.
21. R.J. Light and D.B. Pillemer, Summing Up. The Science of Reviewing Research, (Cambridge, Harvard University Press, 1984): 31.
22. Petitti, 218.
23. Petitti, 129-130.
24. American Medical Association, "Journal of the American Medical Association Instruction for Authors," JAMA, Vol. 277, no. 1 (1997): 74-82.
Go to:
Comments: dqemail@aol.com (10/01/98)