J Manipulative Physiol Ther 2001 (Mar); 24 (3): 170–176 ~ FULL TEXT
Jeffrey R. Cates, DC, MS, David N. Young, DC, PhD,
David J. Guerriero, DC, MS, Warren T. Jahn, DC, MPS et. al.
OBJECTIVE: To review and identify established methods for evaluating the quality of practice guidelines and to use a selected assessment tool to assess 2 chiropractic practice guideline documents.
METHODS: A search of the medical literature was performed to identify current methods and procedures for practice guideline evaluation. Two chiropractic practice guideline documents, Vertebral Subluxation in Chiropractic Practice (CCP) and Guidelines for Chiropractic Quality Assurance and Practice Parameters (Mercy) were then independently evaluated for validity by 10 appraisers using the identified appraisal tool. The appraisal scores were tabulated, and consensus appraisals were generated for the CCP and Mercy guideline documents.
RESULTS: The “Appraisal Instrument for Clinical Guidelines” (Cluzeau instrument) was identified as a reliable and valid method of guideline evaluation. The result of the application of this appraisal tool in the assessment of the CCP and Mercy guideline documents was that the former scored notably lower than the latter. On the basis of the results of the guideline appraisals, the CCP document is not recommended, and its guidelines are not considered suitable for application in chiropractic practice. The Mercy guidelines are recommended for application in chiropractic practice, with the proviso that new scientific data should be considered.
CONCLUSION: The literature reviewed suggests that professional organizations or groups should undertake a critical review of guidelines using available critical guideline appraisal tools. Guideline validity appraisal should be done before acceptance by the chiropractic profession. To avoid unwarranted utilization of poorly constructed guidelines, it is strongly recommended that all future guidelines be reviewed for validity and scientific accuracy with the findings published in a medically indexed journal before they are adopted by the chiropractic community.
From the Full-Text Article:
There has been a prodigious production of practice guidelines
relating to health care issues in recent years. With so
many sets of guidelines being produced by so many organizations,
physicians now find that they need to assess guidelines’
quality and validity before using them in practice. The
question of how to evaluate clinical guidelines then arises.
We will entertain the hypothesis that a method of evaluating
guidelines exists by which chiropractic guideline quality can
be reliably evaluated.
Practice guidelines are expert or evidence-based recommendations
regarding optimal clinical protocols for a health
care field or condition under treatment. Guidelines, which
can be either procedure-based or condition-based, are considered
valid when they optimize patient care and result in
health gain at the expected costs. Numerous guideline documents
regarding chiropractic practice have been developed;
they have varying degrees of overall quality, validity, and
clinical applicability. [1-3] High-quality practice guidelines
clarify what interventions are effective in various circumstances
and specify the extent of the documentation supporting
those interventions. Guidelines can also reveal those
procedures and interventions that are poorly supported by
the scientific literature or require additional research to
demonstrate efficacy. Flawed guidelines can result in ineffective
or inappropriate treatment, overutilization, or poor
clinical outcomes and can harm patients by disseminating
information and advice that is scientifically inaccurate. 
Health care guidelines should be subjected to peer review
and testing by professional organizations or groups before the guidelines’ validation and application in clinical practice
to maximize the benefit and limit any potential harm arising
from the use of flawed guidelines. [5-7]
The development of clinical practice guidelines has
evolved from basing recommendations on expert opinions
to relying on a systematic review of evidence regarding the
efficacy of various procedures.  Inasmuch as guidelines now
rely more on documented evidence and less on opinions, it
has become possible to evaluate the quality of a guideline in
a method similar to meta-analysis. 
It is our intent to identify and evaluate some currently
accepted guideline evaluation techniques and to apply them
to 2 chiropractic guideline documents: Vertebral Subluxation
in Chiropractic (CCP),  produced by the Council on
Chiropractic Practice, and Guidelines for Chiropractic
Quality Assurance and Practice Parameters (Mercy), 
forged at the Mercy Center Consensus Conference. The purpose
of our work is to:
(1) identify an effective evaluation instrument and
(2) use the instrument to evaluate these 2 chiropractic guideline documents. It is hoped that this work will serve to identify a method that can be used to evaluate the quality of these and future chiropractic practice
The CCP guidelines are condition-specific and focus on
the diagnosis and management of the vertebral subluxation.
Within the chiropractic profession, there is debate as to the
significance and existence of the vertebral subluxation. The
CCP guidelines fail to objectively address both the scientific
evidence and the controversy regarding the existence or significance
of a subluxation as a diagnostic entity. It is not
within the intent or scope of the present article to address
this controversial issue, but the view that subluxation
assessment is a valid diagnostic method is questioned in
both the scientific literature1, [34, 35] and those chiropractic
guidelines indexed in the National Guideline Clearinghouse
that have evidence rating systems suited to scientific and
technical areas of practice. [11, 36] Hayward et a  recommend
that readers look for an explanation if new guidelines differ
from those currently in use. It is also noted that those recommendations
based solely on clinical judgment and experience
are most susceptible to bias and self interest.  In the
end, if the underlying evidence is weak, no matter what
degree of consensus is attained, the validity of the guidelines
will be limited.  Lack of methods to assess and report
the quality and strength of scientific evidence considered,
failure to consider relevant data, and over-reliance on clinical
judgments and opinions appear to be the major flaws
associated with the CCP guidelines. In addition, the CCP
guidelines lack a rating system that is suited to scientific
areas of practice; the document thereby prohibits the reader
from assessing the evidence supporting the guideline recommendations.
These flaws account for the poor appraisal
scores noted for the CCP guidelines.
The Mercy guidelines were found to be both procedure-based
and broad in scope and to have generally adhered to
acceptable rigors of development. This was reflected in the
final appraisal scores noted above. The major problem identified
with the Mercy guidelines lies in the age of the document.
Inasmuch as the Mercy document is based on the literature
published before 1993, there are obviously several
years of newer research to be considered. Nevertheless, the
general consensus of the appraisers in this study was that the
large majority of the recommendations found in the Mercy guidelines continue to be supported by the current literature
and that the guidelines are still valid and usable — with the
proviso that current research findings published since the
Mercy guideline’s publication should be taken into consideration.
According to the National Guideline Clearinghouse,
 the Mercy guidelines were reaffirmed in 1999 by
the Commission for the Establishment of Guidelines for
Chiropractic Quality Assurance and Practice Parameters.
The greatest shortcoming of this study is that it is based
on the utilization of an instrument that has not yet been
established as a gold standard. Graham et al  note that currently
there is insufficient evidence to support the exclusive
use of any one appraisal instrument, although the Cluzeau
instrument appears to be the one most well developed.
Although the gold standard validity of the Cluzeau instrument
remains to be established, it is considered the most
valid and utilitarian tool of its kind available at this time.
The appraisal tool was optimized to appraise condition-based
guidelines rather than procedure-based guidelines;
the Mercy scores may thus have been lower than they otherwise
could have been, inasmuch as certain criteria did not
apply well to the procedure-based Mercy document. The
appraisal instrument does require a degree of subjective
assessment, which is unavoidable in any evaluation tool.
Statistically, our results are within the value range of
those of Cluzeau et a  and in their analysis of 60 guidelines.
This fact supports the notion that this guideline
appraisal instrument was applied and functioned as intended
by its authors. Our independent appraisers reached a
remarkable consensus supporting the reliability of this
Ultimately, physicians and groups need to determine the
quality of drafted guidelines, and the Cluzeau instrument
facilitates such analysis. To avoid unwarranted utilization of
poorly constructed guidelines, it is strongly recommended
that the quality of all future guidelines be similarly reviewed
for validity and scientific accuracy with the findings published
in a medically indexed journal before adoption by any
Guidelines are considered valid when they optimize
patient care and result in health gain at the expected costs.
Procedures exist for guideline evaluation, and the utilization
of such procedures can identify problems with guideline
quality. At best, invalid guidelines can waste resources; at
worst, they can be detrimental to patient health.16 Early
identification of guidelines lacking validity is essential so
that professional acceptance of and reliance on flawed data
can be avoided.
The Cluzeau instrument is a valid and useful method of
guideline evaluation. It is recommended that future guidelines
be evaluated for validity by means of an accepted
appraisal tool, such as the one used in this review of the
CCP and Mercy guideline documents.
A review of the results of this evaluation led us to conclude
that the development process of the CCP document
might be flawed partly as a result of bias with respect to the
existence and significance of the subluxation and partly as a
result of a flawed evaluation protocol and poor correlation
of scientific evidence. According to the method of guideline
evaluation used in this study, the CCP guidelines fail to document
that fundamental steps of quality guideline development
were followed. The notably low quality assessment
scores for the CCP guidelines resulted in a general consensus
that they are not suitable for use in chiropractic practice,
and they received a rating of “Not recommended.”
Reviewing the appraisal scores and evidence led us to
conclude that the Mercy guidelines are valid and usable in
chiropractic practice, with the proviso that current credible
literature and research findings published since the Mercy
document publication should be taken into consideration.
On the basis of the results of our appraisal, the Mercy guidelines
are “Recommended with proviso.”