Evaluating the Quality
of Clinical Practice Guidelines

This section is compiled by Frank M. Painter, D.C.
Send all comments or additions to:

FROM:   J Manipulative Physiol Ther 2001 (Mar);   24 (3):   170–176 ~ FULL TEXT

Jeffrey R. Cates, DC, MS, David N. Young, DC, PhD,
David J. Guerriero, DC, MS, Warren T. Jahn, DC, MPS et. al.

OBJECTIVE:   To review and identify established methods for evaluating the quality of practice guidelines and to use a selected assessment tool to assess 2 chiropractic practice guideline documents.

METHODS:   A search of the medical literature was performed to identify current methods and procedures for practice guideline evaluation. Two chiropractic practice guideline documents, Vertebral Subluxation in Chiropractic Practice (CCP) and Guidelines for Chiropractic Quality Assurance and Practice Parameters (Mercy) were then independently evaluated for validity by 10 appraisers using the identified appraisal tool. The appraisal scores were tabulated, and consensus appraisals were generated for the CCP and Mercy guideline documents.

RESULTS:   The “Appraisal Instrument for Clinical Guidelines” (Cluzeau instrument) was identified as a reliable and valid method of guideline evaluation. The result of the application of this appraisal tool in the assessment of the CCP and Mercy guideline documents was that the former scored notably lower than the latter. On the basis of the results of the guideline appraisals, the CCP document is not recommended, and its guidelines are not considered suitable for application in chiropractic practice. The Mercy guidelines are recommended for application in chiropractic practice, with the proviso that new scientific data should be considered.

CONCLUSION:   The literature reviewed suggests that professional organizations or groups should undertake a critical review of guidelines using available critical guideline appraisal tools. Guideline validity appraisal should be done before acceptance by the chiropractic profession. To avoid unwarranted utilization of poorly constructed guidelines, it is strongly recommended that all future guidelines be reviewed for validity and scientific accuracy with the findings published in a medically indexed journal before they are adopted by the chiropractic community.

From the Full-Text Article:


There has been a prodigious production of practice guidelines relating to health care issues in recent years. With so many sets of guidelines being produced by so many organizations, physicians now find that they need to assess guidelines’ quality and validity before using them in practice. The question of how to evaluate clinical guidelines then arises. We will entertain the hypothesis that a method of evaluating guidelines exists by which chiropractic guideline quality can be reliably evaluated.

Practice guidelines are expert or evidence-based recommendations regarding optimal clinical protocols for a health care field or condition under treatment. Guidelines, which can be either procedure-based or condition-based, are considered valid when they optimize patient care and result in health gain at the expected costs. Numerous guideline documents regarding chiropractic practice have been developed; they have varying degrees of overall quality, validity, and clinical applicability. [1-3] High-quality practice guidelines clarify what interventions are effective in various circumstances and specify the extent of the documentation supporting those interventions. Guidelines can also reveal those procedures and interventions that are poorly supported by the scientific literature or require additional research to demonstrate efficacy. Flawed guidelines can result in ineffective or inappropriate treatment, overutilization, or poor clinical outcomes and can harm patients by disseminating information and advice that is scientifically inaccurate. [4] Health care guidelines should be subjected to peer review and testing by professional organizations or groups before the guidelines’ validation and application in clinical practice to maximize the benefit and limit any potential harm arising from the use of flawed guidelines. [5-7]

The development of clinical practice guidelines has evolved from basing recommendations on expert opinions to relying on a systematic review of evidence regarding the efficacy of various procedures. [8] Inasmuch as guidelines now rely more on documented evidence and less on opinions, it has become possible to evaluate the quality of a guideline in a method similar to meta-analysis. [9]

It is our intent to identify and evaluate some currently accepted guideline evaluation techniques and to apply them to 2 chiropractic guideline documents: Vertebral Subluxation in Chiropractic (CCP), [10] produced by the Council on Chiropractic Practice, and Guidelines for Chiropractic Quality Assurance and Practice Parameters (Mercy), [11] forged at the Mercy Center Consensus Conference. The purpose of our work is to:

(1) identify an effective evaluation instrument and

(2) use the instrument to evaluate these 2 chiropractic guideline documents. It is hoped that this work will serve to identify a method that can be used to evaluate the quality of these and future chiropractic practice guidelines.


The CCP guidelines are condition-specific and focus on the diagnosis and management of the vertebral subluxation. Within the chiropractic profession, there is debate as to the significance and existence of the vertebral subluxation. The CCP guidelines fail to objectively address both the scientific evidence and the controversy regarding the existence or significance of a subluxation as a diagnostic entity. It is not within the intent or scope of the present article to address this controversial issue, but the view that subluxation assessment is a valid diagnostic method is questioned in both the scientific literature1, [34, 35] and those chiropractic guidelines indexed in the National Guideline Clearinghouse that have evidence rating systems suited to scientific and technical areas of practice. [11, 36] Hayward et a [15] recommend that readers look for an explanation if new guidelines differ from those currently in use. It is also noted that those recommendations based solely on clinical judgment and experience are most susceptible to bias and self interest. [26] In the end, if the underlying evidence is weak, no matter what degree of consensus is attained, the validity of the guidelines will be limited. [5] Lack of methods to assess and report the quality and strength of scientific evidence considered, failure to consider relevant data, and over-reliance on clinical judgments and opinions appear to be the major flaws associated with the CCP guidelines. In addition, the CCP guidelines lack a rating system that is suited to scientific areas of practice; the document thereby prohibits the reader from assessing the evidence supporting the guideline recommendations. These flaws account for the poor appraisal scores noted for the CCP guidelines.

The Mercy guidelines were found to be both procedure-based and broad in scope and to have generally adhered to acceptable rigors of development. This was reflected in the final appraisal scores noted above. The major problem identified with the Mercy guidelines lies in the age of the document. Inasmuch as the Mercy document is based on the literature published before 1993, there are obviously several years of newer research to be considered. Nevertheless, the general consensus of the appraisers in this study was that the large majority of the recommendations found in the Mercy guidelines continue to be supported by the current literature and that the guidelines are still valid and usable — with the proviso that current research findings published since the Mercy guideline’s publication should be taken into consideration. According to the National Guideline Clearinghouse, [36] the Mercy guidelines were reaffirmed in 1999 by the Commission for the Establishment of Guidelines for Chiropractic Quality Assurance and Practice Parameters.

The greatest shortcoming of this study is that it is based on the utilization of an instrument that has not yet been established as a gold standard. Graham et al [32] note that currently there is insufficient evidence to support the exclusive use of any one appraisal instrument, although the Cluzeau instrument appears to be the one most well developed. Although the gold standard validity of the Cluzeau instrument remains to be established, it is considered the most valid and utilitarian tool of its kind available at this time. The appraisal tool was optimized to appraise condition-based guidelines rather than procedure-based guidelines; the Mercy scores may thus have been lower than they otherwise could have been, inasmuch as certain criteria did not apply well to the procedure-based Mercy document. The appraisal instrument does require a degree of subjective assessment, which is unavoidable in any evaluation tool.

Statistically, our results are within the value range of those of Cluzeau et a [16] and in their analysis of 60 guidelines. This fact supports the notion that this guideline appraisal instrument was applied and functioned as intended by its authors. Our independent appraisers reached a remarkable consensus supporting the reliability of this study’s outcomes.

Ultimately, physicians and groups need to determine the quality of drafted guidelines, and the Cluzeau instrument facilitates such analysis. To avoid unwarranted utilization of poorly constructed guidelines, it is strongly recommended that the quality of all future guidelines be similarly reviewed for validity and scientific accuracy with the findings published in a medically indexed journal before adoption by any professional group.


Guidelines are considered valid when they optimize patient care and result in health gain at the expected costs. Procedures exist for guideline evaluation, and the utilization of such procedures can identify problems with guideline quality. At best, invalid guidelines can waste resources; at worst, they can be detrimental to patient health.16 Early identification of guidelines lacking validity is essential so that professional acceptance of and reliance on flawed data can be avoided.

The Cluzeau instrument is a valid and useful method of guideline evaluation. It is recommended that future guidelines be evaluated for validity by means of an accepted appraisal tool, such as the one used in this review of the CCP and Mercy guideline documents.

A review of the results of this evaluation led us to conclude that the development process of the CCP document might be flawed partly as a result of bias with respect to the existence and significance of the subluxation and partly as a result of a flawed evaluation protocol and poor correlation of scientific evidence. According to the method of guideline evaluation used in this study, the CCP guidelines fail to document that fundamental steps of quality guideline development were followed. The notably low quality assessment scores for the CCP guidelines resulted in a general consensus that they are not suitable for use in chiropractic practice, and they received a rating of “Not recommended.” Reviewing the appraisal scores and evidence led us to conclude that the Mercy guidelines are valid and usable in chiropractic practice, with the proviso that current credible literature and research findings published since the Mercy document publication should be taken into consideration. On the basis of the results of our appraisal, the Mercy guidelines are “Recommended with proviso.”

Return to the GUIDELINES Section


Since 3-19-2001

         © 1995–2018 ~ The Chiropractic Resource Organization ~ All Rights Reserved