Content area
Full Text
Background and Purpose. Assessment of the quality of randomized controlled trials (RCTs) is common practice in systematic reviews. However, the reliability of data obtained with most quality assessment scales has not been established. This report describes 2 studies designed to investigate the reliability of data obtained with the Physiotherapy Evidence Database (PEDro) scale developed to rate the quality of RCTs evaluating physical therapist interventions. Method. In the first study, 11 raters independently rated 25 RCTs randomly selected from the PEDro database. In the second study, 2 raters rated 120 RCTs randomly selected from the PEDro database, and disagreements were resolved by a third rater; this generated a set of individual rater and consensus ratings. The process was repeated by independent raters to create a second set of individual and consensus ratings. Reliability of ratings of PEDro scale items was calculated using multi-rater kappas, and reliability of the total (summed) score was calculated using intraclass correlation coefficients (ICC [1,1]). Results. The kappa value for each of the 11 items ranged from .36 to .80 for individual assessors and from .50 to .79 for consensus ratings generated by groups of 2 or 3 raters. The ICC for the total score was .56 (95% confidence interval=.47-.65) for ratings by individuals, and the ICC for consensus ratings was .68 (95% confidence interval=.57-.76). Discussion and Conclusion. The reliability of ratings of PEDro scale items varied from "fair" to "substantial," and the reliability of the total PEDro score was "fair" to "good." [Maher CG, Sherrington C, Herbert RD, et al. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther. 2003;83:713-721.]
Key Words: Evidence-based medicine, Meta-analysis, Physical therapy, Randomized controlled trials.
Systematic reviews of randomized controlled trials (RCTs) are considered by some authors1-3 to constitute the best single source of information about the effectiveness of health care interventions. Most systematic reviews involve assessment of the quality of the RCTs being reviewed because there is evidence that low-quality studies provide biased estimates of treatment effectiveness. For example, RCTs that are not blinded4,5 or do not use concealed allocation4-6 tend to show greater effects of intervention than RCTs with these features.
Systematic reviews may exclude low-quality studies from the analysis (eg, systematic review by Herbert and Gabriel7),...