Conclusion, Appendixes, Glossary, and Index

Conclusion

Appendix A: Participants at the Unconventional Medical Practices Workshop

Appendix B: Comments of the Panel on Mind-Body Interventions on the National Research Council's Reports on Alternative Medicine

In 1991 the National Research Council (NRC) issued an evaluation of some of the therapies examined herein (Druckman and Bjork, 1991). The NRC in 1988 also reviewed certain human-performance technologies designed to enhance human abilities beyond normal levels, which are also the concern of the Panel on Mind-Body Interventions (Druckman and Swets, 1988). Because the conclusions of the NRC reports differ from our own, and because these reports have been influential in shaping public opinion about the effectiveness and benefits of certain mind-body interventions, we believe it is important to comment on these discrepancies.


We shall focus on the NRC's treatment of meditation, one of the approaches we have closely examined, and parapsychology, an indirectly related area, to illustrate these differences of opinion and describe how they have taken shape.


Meditation


The 1991 NRC report stated, "Overall, our assessment of the scientific research on meditation (primarily, transcendental meditation [TM]) leads to the conclusion that it seems to be no more effective in lowering metabolism than are established relaxation techniques; it is unwarranted to attribute any special effects to meditation alone" (Druckman and Bjork, 1991). The NRC report reached this conclusion by drawing primarily on two previous narrative reviews. One of these, by Holmes, covered less than half the relevant studies on TM available at the time it was prepared (Holmes, 1984). The other, by Brener and Connally (1986), also appears to have ignored much of the available and relevant research.


A meta-analysis by TM researchers Dillbeck and Orme-Johnson on the effects of meditation, published in American Psychologist, came to a different conclusion but was ignored in the NRC report. Their quantitative approach showed that the effect size for TM was more than twice that of resting quietly on basal skin resistance, respiration rate, and plasma lactate (Dillbeck and Orme-Johnson, 1987).


Furthermore, Eppley, Abrams, and Shear, addressing psychological and physiological measures of anxiety, showed that TM typically produces two to three times the reductions in effects of chronic stress compared with other meditation and relaxation techniques (Eppley et al., 1989). Yet the NRC report said "no evidence supports the notion that . . . meditation permits a person to better cope with a stressor."


Meta-analysis allows quantitative analysis of various aspects of the literature. For instance, it allows one to compare the results of studies done by experimenters who are cordial, neutral, and negative toward TM. The Eppley meta-analysis demonstrated that the distribution of effects was normal, indicating that the positive conclusions reached in studies of TM are not the result of selective reporting, and that the NRC's characterization of researchers who are practitioners of meditation as subjectively biased "devotees" is without merit. The Eppley meta-analysis also contradicted the Brener and Connally claim that meditation research suffered from "weak design" by providing quantitative demonstration that the results cannot be accounted for by subject selection, experimenter bias, expectancies, or atmospheric effects.


The NRC report embodies some faulty assumptions about meditation. It expresses the expectation that meditation should "[lower] reactivity to challenge"--that is, to make one less responsive to stressors, perhaps through "distracting a person" or providing a "quiet place." But this is neither the traditional nor the express purpose of TM, which is to achieve "restful alertness, a state of unifying capacity." These misunderstandings may be due to the fact, acknowledged by the NRC, that no one on their committee was personally familiar with the experience of any of the meditation practices they reviewed. The difficulties this created were also acknowledged by the committee: "It seems appropriate to be mindful of the constraints that science, as well as culture, background, and personal life experience, place on how the committee views the field of meditation."_


The most glaring omission in the NRC report is a large database (more than 40 published reports) of societal impact studies on what the TM researchers call the consciousness field. The theory underlying this research is that the field, when supported by a sufficient number of meditators, produces the effects and benefits of meditation in the larger population. This is a nonlocal effect, a type of action-at-a-distance, and the TM researchers describe a correspondence to aspects of quantum nonlocality in their efforts to explain the results of these studies.


On the positive side, the NRC report makes a number of very sensible recommendations for research. In a general observation, they state that "learning to relax and enjoy good feelings may prompt a person to make positive changes in his or her work and personal situation. . . . [I]t may be that meditation and relaxation . . . effect cognitive change."_ Their overall conclusion restates a question about relative efficacy and constitutes an implicit recommendation for more incisive research, but they do not dispute the potential therapeutic effects of meditation broadly defined.


Parapsychology


In its 1988 report the NRC is strongly critical of parapsychology, a field that studies, from an independent perspective, the nonlocal events exemplified in prayer and mental-spiritual healing that we have reviewed earlier. The NRC emphasized their belief that more than 130 years of research have failed to find any evidence of parapsychological phenomena. Because of the relevance of this research to issues addressed by the Panel on Mind-Body Interventions, the literature was examined, revealing impressive evidence in clear disagreement with the NRC's conclusion.


In the December 1989 issue of Foundations of Physics, Radin and Nelson reported the largest meta-analysis of parapsychological findings ever done--a total of 832 studies from 68 investigators, involving the influence of human consciousness on microelectronic systems (Radin and Nelson, 1989). The results: "Radin and Nelson's meta-analysis demonstrates that the . . . results are robust and repeatable. Unless critics want to allege wholesale collusion among more than 60 experimenters or suggest a methodological artifact common to . . . hundred[s of] experiments conducted over nearly three decades, there is no escaping the conclusion that [these] effects are indeed possible" (Broughton, 1991; Jahn and Dunne, 1987).


Meta-analysis has also been applied to research studies in precognition, which typically involve card-guessing by a subject before the targets are even prepared. Honorton and Ferrari found 309 studies in English-language publications by 62 investigators, involving more than 50,000 subjects who participated in nearly 2 million trials. Their findings were as follows:


* Thirty percent of the studies produced statistically significant results (where 5 percent was expected by chance). The odds of this result happening by chance are approximately 1 in 1,024.


* The results could not be explained by the failure of researchers to report negative studies (the "file drawer" effect).


* Studies with the most rigorous methodology tended to produce better results (exactly the opposite of critics' claims).


* The effect size remained constant over the more than 50 years under consideration (Honorton and Ferrari, 1989).


An excellent summary of the techniques of meta-analysis applied to several parapsychological databases was published in 1991 by Jessica Utts in Statistical Science (Utts, 1991).


A charge frequently made about parapsychology and the nonlocal therapies we have examined is that the quality of research in these areas is low or substandard. In its 1988 report, the NRC commissioned psychologist Robert Rosenthal of Harvard University to prepare an evaluation of all the controversial areas of interest to the NRC committee. Parapsychology researcher Richard S. Broughton describes this undertaking:


Rosenthal is widely regarded as one of the world's experts in evaluating controversial research claims in the social sciences and has spent much of his career developing techniques to provide objective assessments of conflicting data. Neither Rosenthal nor his coauthor, Monica Harris, had taken any public position on parapsychology. . . . The report by Harris and Rosenthal determined that the "research quality" of the parapsychology research was the best of all the areas under scrutiny. . . . Incredibly . . . [the] committee chairman . . . asked Rosenthal to withdraw the parapsychology section of his report. Rosenthal refused. In the final document, the Harris and Rosenthal report is cited only in the several sections dealing with nonparapsychological topics; there is no mention of it in the parapsychology section (Broughton, 1991).


The Panel on Mind-Body Interventions believes it is necessary to acknowledge and document our differences of opinion with the NRC reports. At the same time, we do not wish to overemphasize or dwell on these conflicting points of view.


If the field of alternative medicine is to progress, it is vital that any evaluation of mind-body practices be comprehensive, rigorous, and unbiased.


References


Brener, J., and S.R. Connally. 1986. Meditation: Rationales, Experimental Effects, and Methodological Issues. Paper prepared for the U. S. Army Research Institute for the Behavioral and Social Sciences, European Division, Department of Psychology, University of Hull, London.


Broughton, R.S. 1991. Parapsychology: The Controversial Science. Ballantine Books, New York, p. 291.


Dillbeck, M.C., and D.W. Orme-Johnson 1987. Physiological differences between transcendental meditation and rest. American Psychologist 42:879-881.


Druckman, D., and R.A. Bjork, eds. 1991. In the Mind's Eye: Enhancing Human Performance. National Academy Press, Washington, D.C.


Druckman, D., and J.A. Swets, eds. 1988. Enhancing Human Performance: Issues, Theories, and Techniques. National Academy Press, Washington, D.C.


Eppley, K.R., A.I. Abrams, and J. Shear. 1989. Differential effects of relaxation technique on trait anxiety: a meta-analysis. J. Clin. Psychol. 45:957-974.


Holmes, D.S. 1984. Mediation and somatic arousal reduction: A review of the experimental evidence. American Psychologist 39:1-10.


Honorton, C., and D.C. Ferrari. 1989. Future telling: a meta-analysis of forced-choice precognition experiments, 1935-1987. J. Parapsychol. 53:281-308.


Jahn, R.G., and B.J. Dunne. 1987. Precognitive Remote Perception. In Margins of Reality: The Role of Consciousness in the Physical World. Harcourt Brace Jovanovich, pp. 149-191.


Orme-Johnson, D.W., and C.N. Alexander. 1992. Critique of the National Research Council's report on meditation. Manuscript available from the first author. Maharishi International University, Fairfield, Iowa.


Radin, D.L., and R.D. Nelson. 1989. Consciousness-related effects in random physical systems. Foundations of Physics 19:1499-1514.


Utts, J. 1991. Replication and meta-analysis in parapsychology. Statistical Science 4:363-403.

Appendix C: WHO Guidelines for the Assessment of Herbal Medicines

Appendix D: Plant Sources of Modern Drugs

Species Family Type of Drug/Product


Acacia senegal (L.) Willd. Leguminosae Gum acacia


Agathosma betulina (Berg.) Pillans Rutaceae Buchu leaf

(Syn.: Barosma betulina (Berg.)

Bartl. et Wendl. f.)


Ammi majus L. Umbelliferae Xanthotoxin


Ananas comosus (L.) Merr. Bromeliaceae Bromelain


Aralia racemosa L. Araliaceae Aralia extracts


Arctostaphylos uva-ursi (L.)

Spreng. Ericaceae Uva ursi


Atropa belladonna L. Solanaceae Belladonna extract


Avena sativa L. Gramineae Oatmeal Concentrate


Berberis vulgaris L. Berberidaceae Berberine


Calendula officinalis L. Compositae Calendula oil


Camellia sinensis L.

(Syn.: Theasinensis L.) Theaceae Caffeine


Capsicum annuum L. Solanaceae Capsicum oleoresin


C. baccatum L. var pendulum (Willd.)

Eshbaugh Capsicum oleoresin


C. chinense Jacquin Capsicum oleoresin


C. frutescens L. Capsicum oleoresin


Capsicum pubescens R. et P. Solanaceae Capsicum extract


Carica papaya L. Caricaceae Papain


Cassia senna L. (Syn.: C. acutifolia Delile senna leaf C. angustifolia Vahl) Leguminosae Sennosides A + B, senna pods


Catharanthus roseus (L.) G. Don Apocynaceae Leurocristine (vincristine) and incaleukoblastine (vinblastine)


Cephaelis ipecacuanha (Brot.) A. Richard Rubiaceae Ipecac fluid extract, ipecac syrup


Chrysanthemum cinerariaefolium (Trev.) Vis. Compositae Pyrethrins


Cinchona calisaya Wedd. Rubiaceae Quinine, quinidine


C. ledgeriana Moens Quinine, quinidine


C. pubescens Vahl Quinine, quinidine


Cinnamomum camphora (L.) J. S. Presl Lauraceae Camphor


Citrus limon (L.) Burm. f. Rutaceae Pectin


Citrus sinensis (L.) Osbeck Rutaceae Citrus bioflavonoids


Colchicum autumnale L. Liliacae Colchicine


Commiphora abyssinica Engl. Burseraceae Myrrh gum


C. molmol Engl. ex Tschirch Myrrh gum


Digitalis lanata Ehrh. Scrophulariaceae Digoxin lanatoside C , and

acetylgitoxin


D. purpurea L. Digitoxin , and

digitalis whole leaf


Dioscorea composita Hemsl. Dioscoreaceae Diosgenin


D. floribunda Mar. et. Gal. Diosgenin


D. deltoidea Wallich Diosgenin


Duboisia myoporoides R. Br. Solanaceae Atropine hyoscyamine scopolamine


Eucalyptus globulus Labill. Myrtaceae Eucalyptol (cineole) eucalyptus oil


Fagopyrum esculentum Moench Polygonaceae Rutin


Frangula alnus P. Miller

(Syn.: Rhamnus frangula L.) Rhamnaceae Frangula bark


Gaultheria procumbens L. Ericaceae Wintergreen oil


Gelsemium sempervirens (L.) St. Hil. Loganiaceae Gelsemium extract


Glycine max (L.) Merr. Leguminosae Sitosterols


Glycyrrhiza glabra L. Leguminosae Licorice extract


Gossypium hirsutum L. Malvaceae Cottonseed oil


Guarea rusbyi (Britton) Rusby Meliaceae Cocillana extract


Hamamelis virginiana L. Hamamelidaceae Witch hazel extract


Lavandula officinalis P. Miller

(Syn.: L. officinalis Chaix) Labiateae Lavender oil


Linum usitatissimum L. Linaceae Linseed oil


Malus sylvestris P. Miller Rosaceae Pectin


Melaleuca leucadendron L. Myrtaceae Cajeput oil


Mentha arvensis L. Labiatae Menthol


M. piperita L. Peppermint oil


M. spicata L. Spearmint oil


Myristica fragrans Houtt. Myristicaceae Nutmeg oil


Myroxylon balsamum (L.) Harms Leguminosae Tolu balsam


M. balsamum var. pareirae (Royle) Harms

(Syn.: M. pareirae (Royle) Klotzsch) Peru balsam


Olea europaea L. Oleaceae Olive oil


Papaver somniferum L. (Paregoric) Papaveraceae Opium extract codeine, morphine, noscapine, and papaverine (33)


Pausinystalia yohimba Pierre ex Beille Rubiaceae Yohimbine


Physostigma venenosum Balf. Leguminosae Physostigmine (eserine)


Pilocarpus jaborandi Holmes Rutaceae Pilocarpine


Pimpinella anisum L. Umbelliferae Anise oil


Piper cubeba L. f. Piperaceae Cubeb oil


Plantago indica L. Plantaginaceae Psyllium husks


P. ovata Forsk. Psyllium husks


P. psyllium L. Psyllium husks


Podophyllum peltatum L. Berberidaceae Podophyllin


Polygala senega L. Polygalaceae Senega fluid extract


Populus balsamifera L.

(Syn.: P. candicans Ait.,

P. tacamahacca P. Miller) Salicaceae Poplar bud


Prunus domestica L. Rosaceae Prune concentrate


P. virginiana L. Wild cherry bark


Quercus infectoria Olivier Fagaceae Tannic acid


Rauvolfia serpentina (L.) Benth. ex Kurz Apocynaceae Reserpine alseroxylon fraction, powdered whole root


Rauvolfia R. vomitoria Afzel. Deserpidine, reserpine, rescinnamine


Rhamnus purshiana DC. Rhamnaceae Cascara bark, casanthranol, danthron(33)


Rheum emodi Wallich Polygonaceae Rhubarb root


R. officinale Baill. Rhubarb root


R. palmatum L. Rhubarb root


R. rhaponticum L. Rhubarb root


Ricinus communis L. Castor oil, ricinoleic acid


Rosa gallica L. Rosaceae Rose petal infusion


Salix alba L. Salicaceae Saligenin


Sanguinaria canadensis L. Papaveraceae Sanguinaria root


Santalum album L. Santalaceae Sandalwood


Sassafras albidum (Nutt.) Nees Lauraceae Sassafras extract


Serenoa repens (Bartr.) Small Palmae Saw palmetto berries


Sesamum indicum L. Pedaliaceae Sesame oil


Sterculia urens Roxb. Sterculiaceae Sterculia gum (karaya gum)


Strychnos nux-vomica L. Loganiaceae Strychnine


Styrax benzoin Dryand. Styracaceae Benzoin gum


S. paralleloneurus Perkins Benzoin gum


Symphytum officinale L. Boraginaceae Allantoin


Syzygium aromaticum (L.) Merr. Myrtaceae Clove oil et Perry


Theobroma cacao L. Sterculiaceae Theobromine


Thymus vulgaris L. Labiatae Thymol


Urginea maritima (L.) Baker Liliaceae Squill extract


Veratrum viride Ait. Liliaceae Veratrum viride extract, cryptennamine


Zea mays L. Graminae Cornsilk

Appendix E: The 20 Most Popular Asian Patent Medicines That Contain Toxic Ingredients

Appendix F: A Guide for the Alternative Researcher

by Claire Cassidy, Ph.D., Barrie Cassileth, Ph.D., Wayne B. Jonas, M.D.,

Richard Pavek, and Linda Silversmith, Ph.D.


The guidelines in this appendix are provided to assist the alternative researcher. The topics presented were selected from a broader array of methodologies and approaches. There is no intention to be all-inclusive. Topics that were omitted may nevertheless be appropriate tools for conducting alternative research.


General Methodological Guidelines


Research studies on alternative medical therapies should be held to the same rigorous scientific and ethical standards that are applied to research on conventional therapies. The guidelines in this appendix represent a summary of major principles for new investigators as they begin to develop research protocols or grant applications. It is recommended that at least one investigator in each study of alternative medicine be experienced in the therapy or research area to be investigated.


It takes as many years to learn how to conduct good research as to become an accomplished practitioner of alternative medicine. Alternative practitioners who wish to do research need to increase their understanding of good research design, but they should also seek out experienced researchers to guide them as collaborators or resources.


Approaches for conducting research must follow a logical sequence for gathering useful data. Typically, research on a given topic is first exploratory, then descriptive and qualitative, then correlative and comparative, and finally experimental and quantitative. Interviews and surveys are examples of descriptive research or possibly correlative/comparative research; best case series fit the correlative/comparative category; and clinical trials are experimental._


Once a decision has been made that a topic is worthy of investigation and not duplicative of previous work, preliminary or pilot studies (exploratory-descriptive) generally are carried out to determine whether there are any promising effects worthy of further investigation and to detect any negative side effects or practical difficulties. These studies may consist of anecdotal case reports, systematic case studies, or uncontrolled single-group studies. Questions are then formulated for use in controlled comparisons (correlative-comparative) using controls such as the best available "other techniques" or a placebo. A large enough group of patients and sufficient time are necessary to provide enough data to suggest whether the treatment is really working and what conditions seem most practical. If effectiveness is reported, then large studies (experimental-quantitative), such as clinical trials, should be organized to find out whether the earlier observations hold true with a more detailed examination using a greater number of participants.


Whatever the research approach, the following procedure generally applies:


1. Identify the paradigm, model, or pattern and explanatory strategies that underlie the intervention under consideration for testing and evaluation.


2. Carefully develop one or two precise research questions to form the basis of the study. The research questions are crucial because they lead directly to the study's objectives, methods, implications, and so on.


3. Ensure that all components of the research plan relate logically to one another. Research questions, goals, subject groups, therapies (regimens, products, etc.) to be studied, and methodologies must be mutually consistent and appropriate. When conceptualizing study objectives, make them consistent with research questions and assumptions of the intervention; in turn, make the study design (the strategy for conducting the study) consistent with research objectives. For all procedures that are operator dependent, identify the skills training and experience of the operator (e.g., teacher or deliverer of treatment). Clarify the nature of the population to be studied; in particular, identify whether the entry criteria lead the study population to be different from the spectrum of people being treated by practicing clinicians.


4. Conduct a library search and gather a comprehensive collection of previous research in the specific area to be studied. Because of incomplete archiving and indexing, computer database searches are currently inadequate to capture the information needed. It may be necessary to read published articles in their entirety and to speak with representatives of alternative medical organizations to locate some references and information. Literature reviews should be comprehensive and systematic (see the "Guidelines for Conducting Literature Reviews" section below).


5. Explain explicitly the methods used to obtain the literature. Simple citation of publications is not adequate. Literature obtained through library search serves as the basis for the "Background" section of grant applications or manuscripts. Background sections should incorporate accurate, high-quality summary evaluations of existing literature. If a systematic review (see the "Introduction to Systematic Reviews" section) or meta-analysis (see the "Introduction to Systematic Reviews" section) has been conducted to quantitatively evaluate the literature, this point should be noted.


6. Clearly define (not just label) the intervention to be tested or evaluated.


7. Include in the study any special diagnostic or outcome aspects of the alternative medicine practice that can be reliably measured.


8. Thoroughly and objectively document all procedures and events that occur during the research study, from subject accrual through data collection, data analysis, and reporting of results.


9. In clinical research (studies involving humans), include adequate control groups and provide followup of subjects over time, with appropriate monitoring of both the intervention group and the control group.


10. In clinical research, consider and minimize any potential risks to subjects. Along with other required information, these risks must be explained to potential subjects in an informed consent document, provided by the sponsoring institution's human subjects committee or institutional review board.


11. Before research begins, decide and indicate in the research proposal what will be considered sufficient evidence to recommend inclusion of the intervention in clinical practice (if relevant).


12. Where appropriate, use standard comparative outcome measures that will allow the new data to be compared with previous and future information on the same topic.


13. Obtain expert guidance on computerizing and analyzing research data. Biostatistics and computer programming assistance will ensure proper management and analysis of data.


Guidelines for Conducting Literature Reviews


Summary information about previous work in a given field is necessary for grant applications and publications. In addition, literature reviews in and of themselves often are useful additions to the literature.


Overview of Goals of the Review


The literature review must address a clearly focused question. It should specify the particular population, intervention or treatment, subject or diagnostic group, or the like, on which the review will focus. A summary table of all studies included in the review, along with their data, may be appropriate. The review should address a specific and pragmatic issue.


Literature Search


The process of collecting relevant articles must be comprehensive and thorough. The search should use bibliographic databases such as MEDLINE, Science Citation Index, Social Science Citation Index, references from relevant articles, personal communications with authors, and manual searches of databases such as Index Medicus. Note that currently this approach may locate only 25 to 50 percent of articles on alternative medicine because most such articles do not appear in standard medical journals (see the "Research Databases" chapter).


Search methods must be systematic and clearly described. Possible selection bias must be addressed when articles are obtained through personal contact. Negative studies should be described along with others; their exclusion suggests possible bias.


Selection of Articles for the Study


The chosen method for selecting articles must be clear, systematic, and appropriate. Inclusion and exclusion criteria should be preestablished in the form of a protocol to be followed when reviewing articles for inclusion; the selection process should then be followed systematically.


The selection protocol should address major criteria that are relevant to the therapy or system under review, including whether the population is adequately defined, whether the exposure or intervention is clearly described, and whether outcomes are detailed and comparable.


Articles should be reviewed in random order and selected as they meet the preestablished criteria. The reliability of the selection process can be measured by comparing articles collected by at least two independent selectors (expert and nonexpert). The extent of selection disagreement can then be evaluated, and a method can be developed to deal with discordant selections.


Research Quality


The quality of the methodology of each study under review is evaluated according to a single set of standards applied to all studies, whether or not the studies have been published. Literature evaluation must be reproducible. It should be conducted by evaluators who are blind with respect to authors, institutions, and study results. These methods of assessment should be described in the introduction to the literature review.


Combining of Results


Results across studies may be combined only when the studies are adequately similar. Study designs, populations, exposures, outcomes, and direction of effect should be similar enough to warrant combining. If studies are methodologically similar, it is less likely that chance influences their results. Analysis of numerous subgroups matched between studies should be avoided, as spurious statistical significance is likely to result. Comparisons are more likely to be valid if variation in the primary studies is considered when results are combined. Differences in study design and components (e.g., population, exposure or intervention, outcomes) should be addressed. Any nonstatistical criteria used for comparison should be explained.


Meta-Analysis and Systematic Reviews


A statistical review method that combines data from several studies is termed meta-analysis (or statistical meta-analysis). These quantitative analyses, which require similar study samples, interventions, and outcomes, can evaluate the magnitude of treatment effect (percentage risk reduction) and the possibility that the differences were due to chance. Meta-analyses can be used to determine the frequency (i.e., quantity) and the quality of the research method employed in studying a specific factor or issue within a single research field or across several fields of study.


Systematic reviews are another orderly approach to reviewing research literature. Like meta-analysis and other quantitative review methods, systematic reviews use clearly specified methods to avoid the introduction of bias in the selection and interpretation of the research literature being reviewed. Clearly defined criteria for including or excluding specific journals and articles are applied; additional criteria are used to evaluate the quality of the measures applied in the reported research to assess the topic of interest. Systematic reviews differ from meta-analyses in that the studies selected for review need not use strictly similar study samples, interventions, or outcome measures.


For additional information, see the "Introduction to Meta-Analysis" and "Introduction to Systematic Reviews" sections.


Significance of Results


The importance of the results can be determined by calculating an odds ratio (the odds of the effect occurring in the exposure group divided by the odds of the effect occurring in the control or comparison group). The resultant number should be large to have any significance. The results should be reported in a clinically meaningful manner such as the absolute difference or the number needed to treat. The results also should be reproducible and generalizable, with similar effects on different types of subject groups. (The level of significance of results could become a criterion for including studies in an alternative medicine research database; such a database is proposed in the "Research Databases" chapter.)


All clinically important consequences should be considered, including other outcomes from the intervention or treatment; these results should be discussed in the context of those analyzed in the review.


Guidelines for Descriptive and Cross-Cultural Studies Using Qualitative Research Methods_


Overview


Many alternative medical systems and practices derive from other cultures or reflect models of health and dysfunction that differ substantively from those current in conventional medicine. As a result, research on alternative medical systems often is in effect, if not explicitly, cross-cultural. The fundamental issue of cross-cultural research is that people who have different views of what constitutes reality also experience reality differently. This means that questions, concepts, diseases, treatments, and research protocols that "make sense" in one setting may not make sense in another.


Before conventional quantitative techniques can be validly applied to the scientific analysis of alternative medical systems, enough must be known of these systems to understand how their beliefs (conscious and unconscious) and behaviors differ from those of conventional systems. These differences can then be taken into account in research design. Failure to know about and account for differences leads to uninterpretable or inaccurate research, raises the potential for misapplying findings to the care of patients, and violates the criterion of model fit._


Methods for cross-cultural research--adjusting for the existence of different models of reality--are most highly developed in the social sciences, especially anthropology and communications, and have been incorporated into medical outcome studies. These methods are mostly categorized as qualitative, but quantitative versions of some techniques are available. In practice, most cross-cultural descriptive research demands the use of qualitative methodologies or a mixture of qualitative and quantitative techniques.


The focus of qualitative research is the individual practitioner or patient, and the community. This form of research is respondent centered, and researchers must take care not to impose their own assumptions or biases on data collection. Qualitative research requires the use of open-ended research techniques or instruments. The research team should include investigators who have had prior experience with qualitative methods and have produced publications that provide evidence of relevant expertise.


Methodological issues of clarity, validity, and the testing of hypotheses are similar in qualitative and quantitative research (see the "Guidelines for Clinical Trials" section for a summary). Correspondingly, in qualitative research as in quantitative research, concepts are detailed, theory is constructed by the testing of hypotheses, data are collected systematically, and criteria of soundness are applied to design, data collection, and interpretation.


Uses of Qualitative Research


Qualitative research is a body of techniques and assumptions concerning how to gather and analyze complex real-world data so that they can be applied to real-world problems (Bernard, 1993; Denzin and Lincoln, 1994; Marshall and Rossman, 1989). All qualitative research shares a set of assumptions or concepts about the research field (Marshall and Rossman, 1989):


* To find out about people's behavior, it is best to immerse oneself in the actual setting chosen for study.


* The participants in the study have values that researchers must honor.


* The researcher's task is to discover these values and perspectives and how they affect the participants' behavior and experience.


* Research is an interactive process.


* Research relies on people's words, stories, and actions as the primary data.


Accordingly, in qualitative or field research, the investigator has direct contact with research subjects and is directly and personally involved in data collection and analysis, with the aim of generating realistic descriptions and explanations. The choice of data collection methods, sampling procedures, and analytic approaches during the research process evolves into a question-specific research design (Crabtree and Miller, 1992). As data are collected and analyzed, this iterative process affects future decisions for additional sampling, collection, and analysis.


Data collection in field research is accomplished primarily through the use of observation, interviewing, and recordings. The researcher may be required to make relatively "unstructured" observations or "structured" observations that depend on a particular knowledge base. Observation is formalized in many ways, including studying proxemics (how people use space) and kinesics (how people move to communicate), participant observation, and various unobtrusive observational measures in which participants are unaware that they are being observed.


The basic approach for data collection usually consists of interviews with individuals or groups. Focus-group interviews are appropriate in some settings and for some purposes but should not replace individual indepth interviews (McCracken, 1988). Sometimes questionnaires can be administered as interviews. Interviews may be conducted at several levels--unstructured (guided everyday conversation), semistructured (more focused but still open-ended), or structured (like spoken questionnaires). Conversations and events may be recorded with audio or video equipment.


Surveys can be constructed on the basis of interview data and, though not administered in a face-to-face setting, can be personalized by offering respondents opportunities to expand on their answers or to contact the researcher for an interview if they want to say more than the survey form permits.


Qualitative researchers have also developed various projective instruments that elicit respondents' unconscious knowledge and beliefs. For example, anthropologists use card-sort and triad-sort techniques, geographers use "mental map" techniques, and psychologists use various picture-response instruments. Preexisting instruments are rarely appropriate for studies across cultures or medical systems.


Much qualitative research also uses secondary sources, such as films, videotapes, texts, and photos. Historical, proxemic, and content analyses of these materials can reveal the unstated values and assumptions of the producers and participants.


To analyze the data collected, the researcher must develop an organizing system, segment the data accordingly, and then determine connections. If the data do not sort well into the categories first selected, the organizing system must be revised. Connections among the sorted data may be made either statistically or interpretively.


Analytical goals


The goal of any analysis is to bring order to what are often extremely complex data. Qualitative researchers try to discover classes of behavior or responses, themes that guide interpretation of events, and differing patterns of response. The first step is descriptive--simply to disentangle the data. Researchers then try to generalize, that is, to find and name the rules under which a particular result may be expected and to explain why this should be so. Much qualitative research eventually is applied in efforts to improve the quality of life, for example, by delivering health care in ways that make sense to the target population.


To be considered useful, qualitative research must fulfill certain criteria of soundness. It must be clear under which circumstances a particular finding applies and whether a finding works consistently. Another demand is that this research be objective. Traditional criteria, such as reliability and validity (see the "Guidelines for Clinical Trials" section), are applied (Kirk and Miller, 1986). However, some authors have defined different criteria of soundness for qualitative research (Lincoln and Guba, 1985; Marshall and Rossman, 1989):


* Credibility. The conduct of inquiry must enable the subjects of the research to say, "Yes, that question (or that interpretation) sounds right to me." This demand can be met because qualitative research deals directly with research subjects.


* Transferability. A researcher samples a population and makes generalizations about the whole population. If another researcher thinks this generalization applies to a different population, tests it, and finds it to be true, then the criterion of transferability has been met. Note that the underlying concepts are transferred, not the specific data.


* Dependability. Rather than assume that observed events can be replicated (the reliability assumption in quantitative research), qualitative researchers want to be able to account for events as they arise and change. When they do so successfully, the criterion of dependability has been met.


* Confirmability. This criterion is met when the findings of one researcher can be confirmed by another. Qualitative researchers can easily bias their data collection by becoming subjectively involved with the research field; this criterion helps to ensure that excessive subjectivity is not biasing the data, that is, that the data are objective.


Although analytical procedures in qualitative research are not necessarily statistical (as they are in quantitative research), some distinct statistical methods can be applied to qualitative research (Bernard, 1993; Miles and Huberman, 1994); software programs such as Anthropac, Ethnograph, and NUDIST, are available to apply these analyses.


Qualitative Versus Quantitative Methods


Research design often requires a combination of qualitative and quantitative approaches. Qualitative and quantitative research differ in the underlying assumptions that researchers make (Cassidy, 1994). In quantitative research, scientists are likely to detail (and often count) particularities and therefore focus on strategies that limit the view, even if they must do so artificially. The randomly assigned, blinded, controlled clinical trial is an important example of this approach; it is not like the real world, because patients normally do not choose practitioners or treatments randomly, and both practitioner and patient usually know what is going on.


Quantitative methods are useful for answering the following types of questions: How many? How much? How often? What size? What are the measurable associations? What will happen if . . .? Does one variable cause the other? Is A more effective than B? The quantitative approach serves to isolate variables so that their influence on outcome can be separated from other factors that might otherwise cloud the interpretation.


In contrast, qualitative researchers are interested in complexity and pattern--the interactions among variables--and purposely avoid approaches (such as the use of controls) that simplify and focus. Qualitative methods are useful for answering the following types of questions: What is going on? What is the nature of the phenomenon? What variations occur? How does it work? How did something happen? What patterns can be identified? Is the original theory or hypothesis correct? Does the original theory fit other circumstances? What difference does this program or intervention make? Why does this intervention work or not work?


In a real-world medical setting, these questions might address the following issues:


* Differences in therapeutic effectiveness when patients are assigned or freely choose their health care.


* How patient and practitioner interact, and how this interaction affects the medical outcome.


* How the design of the health care delivery setting affects patient or practitioner satisfaction.


* How patients compare care in two different medical systems.


* How patients become acclimated in a new (e.g., alternative) system of medical care.


There is another important difference between qualitative and quantitative approaches. Quantitative research depends on an assumption that a certain commonality or unchangingness underlies how materials interact. This assumption translates to a demand that a hypothesis be tested the "same way" and "as planned" in different research settings. Once the research has begun, the protocol cannot be changed, for doing so introduces new variables that would invalidate the work.


Qualitative research depends on the opposite assumption, namely that the real world always involves flux and change. Qualitative research protocols outline the goal and approaches, but they are based on the assumption--indeed, the expectation--that unpredictable events will occur and that the research protocol can be changed as one means of dealing with these events (Marshall and Rossman, 1989). Such changes do not invalidate the qualitative research so long as researchers recognize that change is necessary, document the reasons, and create a logical means to deal with the novel event.


Qualitative methods can explain the real world of alternative health care delivery. The qualitative approach is an ideal way to elucidate outcomes issues (as in cost and clinical effectiveness studies) and can be used in settings where little is known about a practice and its theory, techniques, practitioners, or users. When qualitative and quantitative methods are linked, researchers are able to gather fruitful data suitable for use in improving the delivery of health care.


Guidelines for Screening Best Cases


Introduction


Many practitioners of unconventional therapies for cancer and other illnesses have not documented the effects of their treatments, yet they claim positive results. A process is needed to screen such claims to determine whether each patient, or case, provides enough information to qualify as part of a best case series and then to determine whether there are enough cases to meet criteria for a best case series.


The guidelines summarized below were adapted from a National Cancer Institute publication (NCI, 1991) produced to assist the development and reporting of best case series for unconventional cancer treatments. These guidelines retain references to cancer therapies, but a similar approach could also be applied to some other unconventional treatments. Applying this simple and reliable best case evaluation system should enable many unconventional therapies to be screened for adequate information. If available information were not found to be adequate, further attempts to evaluate the therapy would be postponed until better information could be obtained._


With sufficient information to create a best case series, cases that meet NCI's criteria (or other designated criteria for other health problems) can be determined. Necessary information includes documentation--using standard measures--of the patient's diagnosis, staging (severity of illness), treatment, outcome, and so on. The procedure for determining adequate best case information includes six steps.


Conclusion


NCI's best case criteria represent a specific and reliable means of uncovering therapies worthy of study. This approach uses a single standard to detail the amount of information available and the response achieved.


This method is used to screen charts for adequate information, estimate clinical response, and evaluate practitioner judgment about clinical response. It provides a systematic method for determining which one or ones of the numerous unconventional approaches to cancer warrant further evaluation through clinical trials. The method is applicable to therapies for other problems besides cancer when appropriate evaluators are available.


Guidelines for Clinical Trials


The following guidelines address major methodological issues relevant to designing and conducting clinical trials. The final guideline addresses how interactions between the subject and the health care practitioner may affect study results.


Model Fit


The basic assumptions about health and disease intrinsic to the system under study should be noted, as should the model for classifying and treating patients by that system. For example, if clinical acupuncture care is under investigation, a description of qi and meridians (see the glossary) and the criteria for patient classification and outcome changes must be presented.


The study population should be selected and classified in a way that reflects the assumptions of the model under consideration. For example, if the study addresses disease outcomes, proper diagnostic categories must be used. If the study involves assumed changes in energy patterns, pulses, or symptoms, patients must be classified according to these criteria from the outset. Outcome measures used must be consistent with these assumptions.


The design and methods to explore the intervention must be selected in a way that is consistent with the model's assumptions and with the objectives of the study. Methodologic goals include efforts to (1) demonstrate any effect, (2) assess relative effects between therapies or therapeutic systems, (3) test the utility of an intervention in actual practice, (4) evaluate a possible mechanism of action, (5) examine an assumption that underlies a practice, (6) examine patient reports of satisfaction and relevant explanatory models, (7) examine practitioner explanations of what happened and why, and (8) examine the character of the practitioner-patient relationship and how it affects the delivery and receipt of care.


The goals of the investigation in relation to the system under study must be clearly delineated in the protocol. The study's title and conclusions should reflect the assumptions of the relevant model and the study goals that were actually investigated.


Hypothesis


Clearly established hypotheses should be contained in the research description or grant application. These should identify or predict the main results so that analyses can test the hypotheses.


Patient Selection Bias


The means by which people are identified and accrued to the study, as well as the numbers of potential subjects who decline participation, must be carefully recorded. For example, did subjects come to the study through advertisements? Were they recruited from clinical practices? By random dialing?


Eligibility and selection (inclusion/exclusion) criteria should be clearly stated. Criteria used to diagnose or classify subjects must be valid and reliable. A reference should be given to document the established reliability of the classification system used. In cancer studies, for example, detailed and specific classifications are established (see the "Guidelines for Screening Best Cases" section).


If no generally accepted classification system exists, the system used in the study must itself be detailed and defended in the methods of the current trial.


Randomization or Matching


Comparison groups are developed through a specific process such as randomization, matching, or stratification. Randomization (or a related procedure) applied to a large enough group should distribute differences in the control and treatment groups in a random fashion. In this way the two groups are "equalized" and made as similar as possible except for the intervention to be studied. The method used to create the comparison group should be clearly described. The method should be balanced at least by age, gender, specific diagnosis and stage of disease, important prognostic factors, and other factors relevant to the particular study.


Control Subjects


To obtain comparative data that will shed light on results found in the treatment (or experimental) group of subjects under study, an appropriate control group is needed. Data from control and treatment group subjects are gathered simultaneously by the researchers. Ideally, the groups are identical except for the treatment or intervention to be studied. However, because no two people are identical in every way that may relate to the illness or therapy to be studied, subjects are randomized or matched.


Blinding


Evaluators of the condition of subjects should be blind with respect to (1) whether subjects receive the intervention or a placebo treatment, (2) how the outcome will be measured, and (3) how results will be analyzed.


Crossover Bias


There should be no dilution or co-intervention, that is, the treatment group should not receive any other therapy or intervention in addition to that evaluated in the study. There should be no contamination, that is, control subjects must not receive the same treatment or one that is similar to the treatment received by the experimental subjects.


Confounding Factors


Possible confounding variables (factors that may influence the study's results) must be addressed adequately. The study groups should be comparable on important prognostic factors. All funding sources should be disclosed, and reports should indicate whether these sources were independent of potential profit from the type of treatment under study.


Sample Size


Estimates of the required number of subjects must be made before the study begins and must be discussed in the research proposal. The statistical basis for selecting the number should be given, and the calculations that led to that number should be described. The research proposal also should provide information about how the researchers plan to attain the desired sample size.


Outcomes and Measurement Errors


Outcome and measurement criteria must be clearly defined and explicit. The validity of the outcome measurements used should be established by references and by verification within the study (against a "gold standard" or parallel outcome measures). The measurement methods used must be sensitive enough to detect the outcome or change to be investigated. All important outcomes must be reported.


The duration of effects must be considered in evaluating outcomes. For example, if subjects of a treatment are crossed over to a control group, consideration must be given to whether they were still experiencing effects from their treatment after the crossover. Statistical mechanisms for handling this type of problem exist.


Loss to Followup


At least 80 percent of subjects brought into the study should be shown to remain with the study long enough for necessary followup to occur. Subjects who withdraw from the study must be fully described. For the study results to be acceptable, subject characteristics (including age, gender, diagnosis, stage of disease, and other important factors) must be similar for those who withdrew and those who remain in the study.


Statistical Methods


Descriptive statistics (data) are presented on all prognostic and outcome factors. Inferential and hypothesis-testing statistics (p-values) are calculated and reported for all major treatment-outcome links. Confidence intervals or probability distributions also are reported for primary treatment-outcome links.


Multiple Measures


When more than one measure, variable, or comparison group is assessed, appropriate analyses are applied. Examples of such analyses include analysis of variance with multiple comparison groups, post hoc analyses, subgroup analyses, multiple hypothesis testing with serial t-or z-tests, and serial dependent measures.


Clinical Significance


Clinical (versus statistical) significance indicates whether research effects are important or meaningful. Patient or physician satisfaction with treatment is an example. Results that achieve statistical significance are not meaningful unless they are also clinically important or meaningful in clinical practice. For example, a very small difference in the effectiveness of two treatments would not be likely to change clinical practice or to influence physicians or patients to adopt the new treatment.


The new treatment should have a low risk of causing direct harm in comparison to the risks of not treating the disease. If risks associated with the treatment are low, the treatment is more likely to be used.


Generalizability


Results cannot be generalized beyond the type of illness or patient studied. Any other studies that addressed the same research questions should be discussed in the protocol. If intervention X is shown to work for patients with diabetes, for example, it cannot also be said to work for people with other illnesses. If intervention Y produces good results in breast cancer, it cannot be claimed to work in lung cancer. Broader generalizability is possible only with very large research projects that include adequate numbers of men and women of different age groups, disease severity categories, and stages of the illness.


As a general guideline, there should be at least 40 people in each group for each treatment-outcome link examined.


Disclosure Issues


The sources of funding for the research should be disclosed, as should any additional sources of funding for the participating investigators when these sources have the potential to influence their work. Reports on the research should indicate whether any of these sources might potentially profit from the type of treatment under study or might profit from an alternative treatment if the treatment under study were discredited.


Patient and Practitioner Beliefs and Interactions


Often in clinical trials, the beliefs of and interactions between investigators and subjects are assumed not to be important, but in alternative medicine these are valid concerns. This guideline addresses such personal considerations.


One consideration is bias, which is not usually intentional in research. The differences that could introduce interference or bias in the conduct of the research should be identified and evaluated. Among these are (1) whether the treatment is delivered in the usual method and style used in health care practice, (2) whether the health care practitioner and patient have expectations about the treatment results, (3) whether the patient has complied with the treatment regimen, and (4) whether interference with normal spontaneity and flexibility in patient-therapist interactions has been avoided or noted.


Utility of the treatment involves the question of whether the treatment, as reported, could be applied by practitioners other than those who participated. The investigators' belief in the efficacy of the treatment should also be assessed, and any idiosyncratic responses or beliefs should be described.


Study subjects must be adequately prepared for their participation. The view of each subject on the need for treatment should be evaluated. For example, does the subject regard the problem as a major or minor condition?


The possibility of transpersonal phenomena should also be considered. Such phenomena might include cultural or spiritual perceptions of the study's importance; cultural disparities in treatment delivery; events that might affect outcome, such as direct observer and evaluation effects_; and possible field--that is, nonlocal--effects.


Introduction to Outcomes Research


Outcomes research evaluates the ultimate effects of treatment systems on patients. This evaluation usually involves a retrospective examination of records or databases accumulated by health care practitioners, hospitals, insurers, and government health programs in order to identify which medical interventions produced the best outcomes (Wennberg, 1990). It is also possible to conduct prospective research by tracking clinical practices concurrently into the future. Outcomes research has been described as the use of natural experiments to find what works in medicine.


The databases under examination in outcomes research may be developed by using various kinds of research methods--descriptive (qualitative), best case (mixed qualitative and quantitative), or quantitative. Clinical case records and insurance claims data are often perused.


Advocates of outcomes research claim that it is potentially cheaper and faster than clinical trials and can provide data on treatments that would not otherwise be evaluated. In fact, retrospective database analysis may be the only way to obtain data on treatments with rare complications. Outcomes research is also useful when dealing with "soft" results such as effects on the quality of life. Consequently, some advocates of alternative medical practices consider outcomes research ideal for examining aspects of alternative medicine.


Outcomes research has other inherent advantages. It does not interfere with the doctor-patient relationship, does not require informed consent or permission from an institutional review board (as do clinical trials), and includes groups (such as the elderly, children, the poor, and minorities) that might not be widely represented in clinical trials.


Critics point out that any research based on retrospective analysis of clinical records is flawed by hidden biases in the data. They claim that researchers cannot correct for the subtle reasons why doctors choose one treatment over another for a given patient (or why patients choose their doctors). Furthermore, the records under examination were made for a different purpose and are likely to be incomplete in describing all relevant conditions that may affect the patients whose records are being analyzed.


Proponents and opponents of outcomes research agree that some aspects of the research are useful--that it is important to learn what doctors are actually doing in clinical practice and that this knowledge can provide a basis for further studies, including clinical trials.


One government agency, the Agency for Health Care Policy and Research (AHCPR), was created in 1989 largely to conduct outcomes research. However, in a recent article in Science, Anderson (1994) reported that "after spending nearly $200 million on outcomes research (about one-third of the agency's budget . . .), AHCPR cannot point to a single case in which its database studies have changed general clinical practice." Anderson further noted that even the agency's most definitive result--a guideline to physicians that "watchful waiting" is more appropriate for some patients than surgery for benign prostate disease (see the "Research Methodologies" chapter)--was accompanied by a recommendation for a clinical trial to confirm these findings.


Increasingly, it appears that AHCPR will use its database analyses of outcomes to supplement and complement other tools, including case control studies, meta-analyses of previous studies, and clinical trials. Two new references are expected to help researchers rank the value of outcomes research: (1) the proceedings of a March 1993 conference sponsored by the New York Academy of Sciences that analyzed the relative merits of outcomes research and clinical trials (Warren and Mosteller, 1994); and (2) the results of an 18-month study by the Office of Technology Assessment (OTA) analyzing AHCPR's outcomes research (publication due September 1994)._


Introduction to Meta-Analysis


The term meta-analysis was first coined by G.V. Glass, in a 1976 study of the efficacy of psychotherapy, as "the statistical analysis of a large collection of results from individual literature, for the purpose of integrating the findings." Although meta-analytic procedures have been widely employed in the social sciences since the early 1970s, many did not consider it a valid tool for the natural sciences until numerous retrospective studies accumulated that used meta-analysis to analyze data that had previously been studied with other statistical tools. As these studies illustrated both the statistical power and the increased information provided by meta-analysis, interest in its medical applications began to increase significantly. Since then, meta-analysis has been applied to questions of efficacy (e.g., chemotherapy in breast cancer, patient education interventions in clinical medicine, spinal manipulation); questions of cause and effect (e.g., effect of exercise on serum lipid level); and, increasingly, public health problems. Today meta-analysis is being used in a variety of settings to draw conclusions from results collected from literature or narrative reviews and from data pooled from independent studies (often clinical trials).


In general, meta-analysis is a systematic method that uses statistical analysis for extracting, comparing, and combining results from independent studies to obtain quantifiable outcomes. Meta-analysis also can help detect gaps in knowledge in the published literature and thus can help provide guidance for future research. Although there have been several approaches to meta-analysis, each follows the same basic procedure:


1. Define the problem and criteria for admission of studies.


2. Locate research studies.


3. Classify and code study characteristics.


4. Measure study characteristics quantitatively on a common scale.


5. Aggregate findings to study characteristics (analysis and interpretation).


6. Report the results.


Problem formulation includes explicit definition of outcomes and potentially confounding variables. Carefully done, this step enables the investigator to focus on the relevant measures in the studies under consideration and to specify the relevant methods for classifying and coding study characteristics. The literature search uses a systematic approach to locating studies. First, information is obtained from colleagues in a particular discipline. Second, the various indexes, abstracting services, and electronic databases are searched. Third, references from the primary articles are used to find secondary sources of information. Finally, information is gathered from academic, private, and government sources, including unreferenced reports and unpublished data.


In order to measure results across disparate studies, several methods are used. The most common method is to measure the effect size (i.e., an index of both the direction and the magnitude of the effect of a procedure under study). One estimate of the effect size for quantitative data is the difference between the two group means, divided by the control group standard deviation, (Xt-Xc)/Sc, where Xc is the mean of the control group and Sc is the standard deviation of the control group. Effect size expresses differences in standard deviation units so that, for example, if a study has an effect of 0.2 standard deviation units, the overall effect size is only half that of another study that has an effect size of 0.4 standard deviation units. The appropriate measure of effect across the research literature varies according to both the nature of the problem being assessed and the availability of published data. Pooling of data from controlled clinical trials, for example, has been more widely used in the medical literature than for other subjects.


Effect size for proportions has been calculated in cohort literature as either a difference, Pt-Pc, or as a ratio, Pt/Pc. The latter has the advantage of considerable change relative to the control percentage; in epidemiological studies, it is equivalent to the concept of risk ratio.


Whatever combination statistic is used, a systematic quantitative procedure to accumulate results across studies should include the following:


1. Summary descriptive statistics across studies, and the averages of those statistics.


2. Calculation of variance of a statistic across studies.


3. Correction of the variance by subtracting sampling error.


4. Correction in the mean and variance for study artifacts other than sampling, such as measurement error.


5. Comparison of the corrected standard deviation to the mean to assess the size of the potential variation across studies.


The value of meta-analysis is that as evidence begins to accumulate, meta-analysis forces systematic thought about methods, outcomes, categorizations, populations, and interventions. In addition, it offers a mechanism for estimating the magnitude of the effect in terms of a statistically significant effect size or pooled odds ratio. Furthermore, the combination of data from several studies increases generalizability and potentially increases statistical power, thus enabling more complete assessment of the impact of a procedure or variable. Quantitative measures across studies also can give insight into the nature of the relationships among variables and can provide a mechanism for detecting and exploring apparent contradictions in research results. Further, because meta-analysis is less subjective than other analytical methods, it has the potential to decrease investigator bias.


However, like the value of all review methods, the value of meta-analysis can be limited by a number of factors. For example, the current use of parametric statistical methods for meta-analysis is the subject of intense theoretical study. Other methodological issues of concern include bias, variability between studies, and the development of models to measure variability across studies. One major concern about qualitative reviews of the literature is that although meta-analysis is more explicit, it may be no more objective than a narrative review. Both critics and advocates of meta-analysis are concerned that an unwarranted sense of scientific validity, rather than true scientific understanding, may result from quantification. More simply stated, use of sophisticated statistics will not improve poor data but could lead analysts to an unwarranted level of comfort with their conclusions.


Introduction to Systematic Reviews


The systematic review is an orderly approach to reviewing research literature that minimizes the problems that can arise with less scientifically rigorous review methods (Larson et al., 1992). To avoid introducing bias in the selection and interpretation of the literature under study, systematic reviews spell out in advance the approach to be taken. Systematic review entails defining criteria for (1) the selection of journals and articles to include and exclude, (2) the quality of the measures used in the selected literature to assess the factor being reviewed, and, (3) the quality of each study's research methodology. The technique also looks at the frequency of assessment of a particular research question, variable, or measure.


Advantages


Systematic reviews, like meta-analyses and unlike standard literature reviews, are replicable from one reviewer to the next. This point is particularly important when a potentially controversial research topic is being evaluated.


Systematic reviews differ from meta-analytical reviews in two major ways. First, the systematic review costs much less--only 10 to 20 percent of the expense of a similarly sized meta-analysis. Second, systematic reviews can consider single factors of interest within an inadequately developed research field. In contrast, meta-analyses require a well-developed research field with a large amount of experimental or quasi-experimental research; they also require that an adequate number of studies address essentially the same research question using comparable study samples.


While systematic reviews can examine the key or central findings in studies, they also permit analysis of noncentral or peripheral factors. Thus systematic reviews are particularly useful in examining an underdeveloped or infrequently studied research issue.


Method


There are five key steps in conducting a systematic review: (1) selecting the factor or factors to be studied; (2) deciding whether to use an exhaustive review or field review approach; (3) assessing the frequency and quality of measurement of the factor of interest; (4) evaluating the studies that contain the factor of interest; and (5) determining and maintaining reviewer reliability.


Selecting the factor or factors to be studied. This first step involves formulating research questions based on the topic the reviewer wishes to study. Each systematic review should address clear research questions. For example, several systematic reviews have focused on whether the quantity or quality of research containing religious variables was substandard in certain clinical scientific literatures (Larson et al., 1986). Another review concerning the effects of pornography asked whether existing research demonstrated harm--or lack of harm--in assessing the associations in each literature report between exposure to pornographic materials and changes in attitudes concerning rape or aggression toward women.


Deciding whether to use an exhaustive review or field review approach. Both types of systematic reviews use research reports that have undergone a peer review process of critique and revision prior to being published. However, criteria for what to include and exclude are defined differently for the two types of reviews.


The exhaustive review method involves identifying every possible peer-reviewed study from every relevant field of study that includes information about the factor of interest. This review is carried out in three steps. (1) First, an initial list of articles is prepared, based on a multiple, overlapping, computerized literature search that uses multiple key-word terms and indexes. (2) Next, other potentially relevant articles are identified in the reference sections of the articles obtained in the initial search, and these new articles are also searched for relevant references. This repeated reference review continues until no new articles can be identified for addition to the master list. (3) The final step is the circulation of the list of articles to identified experts, such as the three to five researchers with the most publications on the research study list; these researchers are asked to identify additional relevant articles.


In contrast, field reviews involve selecting only one field of study, the leading peer-reviewed journals in that field, and the period to be reviewed (usually 5 to 10 years). The leading journals are identified as the ones most frequently cited in a particular research field, by using the Science Citation Index or the Social Science Citation Index as a citation source. (These indexes provide ratings of journals in various research fields based on the frequency with which their articles are cited). If the goal is to define the most accurate and up-to-date research in a specific field, then the field review is the more appropriate type of systematic review to use.


The field reviewer obtains a proper sample by manually searching through every journal issue and every article in the journal to identify studies that include the review factor of interest. Some topics of previous systematic reviews include mental health factors in nursing home studies, AIDS research in general medical journals, and religious factors in psychiatry, family medicine, and pastoral care journals. The total numbers of articles scanned and articles selected should be tracked. Editorial articles, commentaries, and other nonquantifiable opinion articles should be excluded.


Assessing the frequency and quality of measurement of the factor of interest. In this step the factor of interest is examined across the reviewed articles to determine whether it is of major or minor importance--that is, whether it is frequently or infrequently assessed. Additional information is tabulated concerning whether the factor is being assessed through use of one or several questions and--if through several questions--whether reliability was reported or demonstrated.


Evaluating the studies that contain the factor of interest. Next the research quality of the studies that include the factors of interest is assessed. If a study is poorly designed, its findings may be questionable.


Assessing the quality of the methods used requires clearly defining each study factor, including such variables as the response rate, size of the study population, use of a control or comparison population, type of sampling method used, and whether study measures demonstrated reliability. For example, defining the response rate might entail grouping rates in categories: low, less than 50 percent; medium, 50 to 69 percent; and high, 70 percent and more. Similarly, other factors require some definition and grouping.


Determining and maintaining reviewer reliability. Reproducibility of systematic reviews depends on training multiple reviewers to appropriately assess the factors of interest. The goal here is statistical reliability, so that reviewers reviewing the same articles achieve the same assessments. Training reviewers has been found to produce replicable results with reliabilities above 0.90 (Larson et al., 1992).


High reliability can be maintained through periodic checks--especially if a large number of studies and a large number of reviewers are involved--and, if necessary, retraining of reviewers.


Usefulness


The kinds of information that systematic reviews can provide about a specific research field or topic include the following:


* Number of studies assessing the factor of interest.


* Statistical reliability of measures assessing the factor of interest.


* Approach most often used for assessing the factor of interest.


* Frequency of assessing the factor as a variable of major versus minor study relevance.


* Quality of the research studies that include the factor of interest.


Selected Bibliography for Researchers


Abramson, J.H. 1988. Making Sense of Data. A Self-Instruction Manual on the Interpretation of Epidemiologic Data. Oxford University Press, Oxford.


Aldridge, D. 1987. Clinical assessment of acupuncture in asthma therapy: discussion paper. J. Royal Soc. Med. 80:222-224.


Aldridge, D. 1989. A guide to preparing a research application. Comp. Med. Res. 3(3):31-37.


Altman, D.G. 1991. Practical Statistics for Medical Research. Chapman & Hall, London.


Anderson, C. 1994. Measuring what works in health care. Science 263:1080-1082.


Anthony, H.M. 1987. Some methodological problems in the assessment of complementary therapy. Stats. Med. 6:761-771.


Anthony, H.M. 1989. Clinical research: questions to ask and the benefits of asking them. Comp. Med. Res. 3(3):3-6.


Bailar, J.C., and K. Patterson. 1986. Journal peer review: the need for a research agenda. In J.C. Bailar III, and F. Mosteller, eds. Medical Uses of Statistics. NEJM Books, Waltham, Mass.


Battista, R.M., and S.W. Fletcher. 1988. Making recommendations on preventive practices: methodological issues. Am. J. Prev. Med. 4 (suppl):53-67.


Bauer, H.H. 1992. Scientific Literacy and the Myth of the Scientific Method. University of Illinois Press, Urbana, Ill.


Baum, M. 1989. Rationalism versus irrationalism in the care of the sick: science versus the absurd. Med. J. Aust. 151:607-608.


Baum, M. 1991. Rationalism versus irrationalism in the treatment of cancer--quack cures or scientific remedies? Surgery 1:2223a-2223c.


Bensoussan, A. 1991. Contemporary acupuncture research: the difficulties of research across scientific paradigms. Am. J. Acupunc. 19(4):357-365.


Bernard H.R. 1993. Research Methods in Cultural Anthropology (2nd ed.). Sage Publication, Newbury Park, Calif.


Bracken, M.B. 1987. Clinical trials and the acceptance of uncertainty. BMJ 294:1111-1112.


Brink, P., and M. Wood. 1988. Basic Steps in Planning Nursing Research (3rd ed.). Jones & Bartlett, Boston.


Briscoe, M.E. 1990. A Researcher's Guide to Scientific and Medical Illustrations. Springer-Verlag, New York.


Canter, D. 1987. A research agenda for holistic therapy. Comp. Med. Res. 2:104-121.


Cassidy, C. 1994. Unraveling the ball of string: reality, paradigms, and the study of alternative medicine. Advances, the Journal of Mind-Body Health 10:5-31.


Cassileth, B.R. 1984. Contemporary unorthodox treatments in cancer medicine. Ann. Intern. Med. 101:105-112.


Cassileth, B.R., E.J. Lusk, D. Guerry, et al. 1991. Survival and quality of life among patients on unproven versus conventional cancer therapy. N. Engl. J. Med. 324:1180-1185.


Chalmers, T.C., P. Celano, H.S. Sachs, et al. 1981. Bias in treatment assignment in controlled clinical trials. N. Engl. J. Med. 309:1358-1361.


Chalmers, T.C., H.J. Smith, B. Blackburn, et al. 1981. A method for assessing the quality of a randomized control trial. Controlled Clin. Trials 2:31-49.


Colditz, G.A., J.N. Miller, and F. Mosteller. 1989. How study design affects outcomes in comparisons of therapy. I: Medical. Stat. Med. 8:441-454.


Coulter, H.L. 1991. The Controlled Clinical Trial: An Analysis. Center for Empirical Medicine, Washington, D.C.


Crabtree, B.F., and W.L. Miller, eds. 1992. Doing Qualitative Research. Sage Publications, Newbury Park, Calif.


Crichton, N.J. 1990. The importance of statistics in research design. Comp. Med. Res. 4(2):41-49.


Denzin, N., and Y. Lincoln, eds. 1994. Handbook of Qualitative Research. Sage Publications, Thousand Oaks, Calif.


DerSimonian, R., L.J. Charette, B. McPeek, et al. 1982. Reporting on methods in clinical trials. N. Engl. J. Med. 306:1332-1337.


Detsky, A.S., C.D. Naylor, K. O'Rourke, et al. 1992. Incorporating variations in the quality of individual randomized trials into meta-analysis. J. Clin. Epidemiol. 45(3):255-265.


Diamond, G.A., and T.A. Denton. 1993. Alternative perspectives on the biased foundations of medical technology assessment. Ann. Intern. Med. 118:455-464.


Dickerson, K. 1990. The existence of publication bias and risk factors for its occurrence. JAMA 263:1385-1389.


Druckman, D., and R.A. Bjork, eds. 1991. In the Mind's Eye: Enhancing Human Performance. National Academy Press, Washington, D.C.


Easterbrook, P.J., J.A. Berlin, R. Gopalan, et al. 1991. Publication bias in clinical research. Lancet 337:867-872.


Eddy, D.M. 1990. Should we change the rules for evaluating medical technologies? In A.C. Gelijns, ed. Modern Methods of Clinical Investigation? National Academy Press, Washington, D.C.


Eddy, D.M. 1992. Assessing Health Practices & Designing Practice Policies: The Explicit Approach. American College of Physicians, Philadelphia.


Eddy, D.M., V. Hasselblad, and R. Shachter. 1992. Meta-Analysis by the Confidence Profile Method. Academic Press, Boston.


Eisenberg, D.M., R.C. Kessler, C. Foster, et al. 1993. Unconventional medicine in the United States--prevalence, costs, and patterns of use. N. Engl. J. Med. 328:246-252.


Emerson, J.D., E. Burdick, D.C. Hoaglin, et al. 1990. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Controlled Clin. Trials 11:339-352.


Ernst, E., T. Saradeth, and K.L. Resch. 1993. Drawbacks of peer review. Nature 363:296.


Feinstein, A.R. 1983. An additional basic science for clinical medicine: II. The limitations of randomized trials. Ann. Int. Med. 99:544-550.


Feinstein, A.R. 1985. Clinical Epidemiology: The Architecture of Clinical Research. W.B. Saunders, Philadelphia.


Fisher, P. 1990. Research into homeopathic treatment of rheumatological disease: why and how? Comp. Med. Res. 4(3):34-40.


Flay, B.R. 1986. Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev. Med. 15:451-474.


Fricke, R., and G. Treinis. 1985. Einfhhrung in die Metaanalyse. Verlag Hans Huber, Berlin.


Fuchs, V.R., and A.M. Garber. 1990. The new technology assessment. N. Engl. J. Med. 323(10):673-677.


Ganiats, T.G. 1993. Are all outcomes created equal? Fam. Pract. Res. J. 13(1):1-5.


Gehan, E.A. 1982. Progress of therapy in acute leukemia:1948-1981. Controlled Clin. Trials 3:199-207.


Gelband, H., et al. 1990. Unconventional Cancer Treatments. U.S. Government Printing Office, Washington, D.C.


Gelijns, A.C., and S.O. Thier. 1990. Medical technology development: an introduction to the innovation-evaluation nexus. In A.C. Gelijns, ed. Modern Methods of Clinical Investigation. National Academy Press, Washington, D.C.


Gerbarg, Z.B., and R.I. Horwitz. 1988. Resolving conflicting clinical trials. J. Clin. Epidemiol. 41(5):503-509.


Gevitz, N., ed. 1988. Other Healers: Unorthodox Medicine in America. Johns Hopkins University Press, Baltimore.


Glass, E.V. 1976. Primary, secondary, and meta-analysis of research. Educational Researcher. 5:3-8.


Greenberg, R.P., R.F. Bornstein, M.D. Greenberg, et al. 1992. A meta-analysis of antidepressant outcome under "blinder" conditions. J. Consult. Clin. Psychol. 60(5):664-669.


Guyatt, G.H., J.L. Keller, R. Jaeschke, et al. 1990. The n-of-1 randomized control trial: clinical usefulness--our three-year experience. Ann. Intern. Med. 112: 293-299.


Haley, R.W. 1994. Designing clinical research. In Y.C. Pak and P.M. Adams, eds. Techniques of Patient Oriented Research. Raven Press, New York.


Hauser, S.P. 1991. Unproven methods in oncology. Eur. J. Cancer 27(12):1549-1551.


Haynes, R.B. 1991. ACP Journal Club's Modus Operandi [Editorial]. Ann. Intern. Med. 115, suppl 3:A14.


Heron, J. 1986. Critique of conventional research methodology. Comp. Med. Res. 1(1):10-22.


Hill, C., and F. Doyon. 1990. Review of randomized trials of homeopathy. Rev. Epidemiol. 38:139-142.


Hinkle, L.E., and H.G. Wolff. 1958. Ecological investigations of the relationship between illness, life experiences and the social environment. Ann. Intern. Med. 49:1373-1388.


Hornung, J., and K. Linde. 1991. Guidelines for the exact description of the preparation and mode of application of serial dilutions and potencies on ultra low dose effects and homeopathic research--a proposal. Berlin Journal of Research in Homeopathy 1(2):121-123.


Hufford, D.J. 1988. Contemporary folk medicine. In N. Gevitz, ed. Other Healers: Unorthodox Medicine in America. Johns Hopkins University Press, Baltimore.


Hulley, S.B., and S.R. Cummings, eds. 1988. Designing Clinical Research. Williams & Wilkins, Baltimore.


James, I. 1989. Tactics and practicalities. Comp. Med. Res. 3(3):7-10.


Jenicek, M. 1989. Meta-analyses in medicine--where we are and where we want to go. J. Clin. Epidemiol. 42(1):35-44.


Jingfeng, C. 1987. Toward a comprehensive evaluation of alternative medicine. Soc. Sci. Med. 25(6):659-667.


Jonas, W.B. 1992. Evaluation of studies involving non-mainstream medicine. Presented at the 7th Annual Primary Care Research and Statistics Conference, University of Texas Health Science Center, San Antonio.


Kiene, H. 1993. Kritik der klinischen Doppelblindstudie. MMV Medizin Verlag, Mhnchen.


Kirk, J., and M. Miller. 1986. Reliability and Validity in Qualitative Research. Sage Publications, Newbury Park, Calif.


Kleijnen, J. 1991. Food supplements and their efficacy. CIP-Gegevien Koninklijke Bibliotheek, Den Haag, Netherlands.


Kleijnen, J., and P. Knipschild. 1992. Review articles and publication bias. Arzneimittelforschung 42:587-591.


Kleijnen, J., P. Knipschild, et al. 1991. Clinical trials of homeopathy. BMJ. 302:316-323.


Kleijnen, J., G. ter Riet, and P. Knipschild. 1991. Acupuncture and asthma--a review of controlled trials. Thorax 46(11):799-802.


Kleinman, A., L. Eisenberg, and B. Good. 1978. Culture, illness, and care. Ann. Intern. Med. 88:251-258.


Knipschild, P., J. Kleijnen, et al. 1990. Zur Glaubwirdigkeit alternativer Medizin. Skeptiker 3(3):4-8.


L'AbbJ, K.A., A.S. Detsky, and K. O'Rourke. 1988. Meta-analysis in clinical research. Ann. Intern. Med. 107:224-233.


Larson, D.B., L.E. Pastro, J.S. Lyons, et al. 1992. The systematic review: an innovative approach to reviewing research. Paper prepared for the Department of Health and Human Services, Assistant Secretary for Planning and Evaluation, by the Family and Community Policy Division.


Larson, D.B., E.M. Pattison, D.G. Blazer, et al. 1986. Systematic analysis of research on religious variables in four major psychiatric journals, 1978-1982. Am. J. Psychiatry 143:329-334.


Leede, P.D. 1992. Practical Research: Planning and Design (5th ed.). McMillan, New York.


Leibrich, J. 1990. Measurement of efficacy: a case for holistic research. Comp. Med. Res. 4(1):21-25.


Lewitt, G.T., and D. Aldrich. 1993. Clinical Research Methodology for Complementary Therapies. Holder & Stoughton, London.


Lincoln, Y., and E. Guba. 1985. Naturalistic Inquiry. Sage Publications, Beverly Hills, Calif.


Little, J.M. 1993. Eupompus gave splendour to art by numbers. Lancet 341:878-880.


Lyne, N. 1989. Theoretical and empirical problems in the assessment of alternative medical technologies. Scand. J. Soc. Med. 37:257-263.


Marshall, C., and G. Rossman. 1989 Designing Qualitative Research. Sage Publications, Newbury Park, Calif.


McCracken, G. 1988. The Long Interview. Sage Publications, Newbury Park, Calif.


Miles, M., and H. Huberman. 1994. Qualitative Data Analysis (2nd ed.). Sage Publications, Thousand Oaks, Calif.


Moher, D., et al. 1993. Proceedings of Assessing the Quality of Randomized Controlled Trials (RCTs): The Development of a Consensus. Ottawa, Ontario, Canada, October 7-8, 1993.


Mulow, C.D. 1987. The medical review article: state of the science. Ann. Int. Med. 106:485-488.


Murray, R.H., and A.J. Rubel. 1992. Physicians and healers: unwitting partners in health care. N. Engl. J. Med. 326:61-64.


National Cancer Institute. 1991. Preparation of Best Case Series and the Conduct of Pilot Clinical Trials Using Unconventional Cancer Treatments. (Informal publication, available from