Traditionally, when the effect of an intervention estimated from a clinical trial was published, it was reported with a p-value. Many statistical authorities have recommended reporting the effect of an intervention with a 95 % confidence interval instead of with a p-value. Such authorities include the Cochrane Collaboration,1 the British Medical Journal,2 the American Statistical Association,3 the journal Statistics in Medicine,4 and proponents of evidence-based practice in medicine and physical therapy.5–7

Given how widely p-values are used in the reporting of clinical trials,8,9 it may surprise some readers that statistical authorities recommend an alternative approach. However, null hypothesis statistical tests, which generate p-values, have numerous problems.8,10 First, the statistical significance of null hypothesis statistical tests has poor replicability. If a study generated a p-value between 0.005 and 0.05, then replicating the study with a new random sample from the same clinical population would have a 33 % chance of a non-significant result.10 Second, p-values do not provide evidence about the precision of the estimate derived from the study. In contrast, confidence intervals distinguish between important differences that p-values do not. For example, when the p-value is non-significant, it may be that (a) the intervention is ineffective and the study has generated a precise estimate of that, or (b) the study was underpowered and the effect remains uncertain. Confidence intervals help to distinguish between those two scenarios. Furthermore, even when a p-value is significant, confidence intervals can distinguish how precisely the study has estimated the magnitude of the intervention's effect and can contribute to interpretation of whether the intervention is clinically worthwhile. P-values don't provide this information.7,11

The physical therapy profession has made efforts to increase the use of confidence intervals, including in publications in the 2000s,12 2010s,11,13,14 and recently.7 An analysis of a representative sample of physical therapy trials showed a modest but steady increase in the use of confidence intervals over a 30-year period.9

Although the trends showing an increase in the use of confidence intervals are heartening, there are several reasons why it is important to continue to monitor the use of confidence intervals in physical therapy trials. First, the percentage of physical therapy trials reporting confidence intervals at the end of the 30-year analysis period (2016) had only reached 42 %,9 so further improvement is needed. Second, in some professions and journals, the proportion of studies reporting confidence intervals has plateaued.15,16 Third, in 2022, there was an initiative by the International Society of Physiotherapy Journal Editors to encourage routine use of confidence intervals instead of p-values,7 so establishing a baseline prevalence immediately prior to that initiative will help to later discern its effect.

To get the most benefit from a confidence interval, researchers must do more than simply report it. Researchers must also interpret the relevance of the range of values within the confidence interval and consider the implications arising from them. Many researchers calculate confidence intervals at the request of editors, but then ignore them and interpret their trial's result dichotomously as statistically significant or non-significant instead.17 Interpretation is crucial but no study of physical therapy trials has yet examined whether confidence intervals are interpreted when they are reported.

It is common that clinical trials present underpowered samples due to several factors, including small sample size, unexpected low inflow of patients, or high attrition rates. Confidence intervals can be useful to describe the effect of sample size and indirectly reflect the statistical power of the clinical trial, which is influenced by the sample size. Although some issues have been raised regarding common approaches to sample size calculation in clinical trials,10 if researchers are going to do a sample size calculation then it also seems worth observing how often the intended sample size is achieved in trials of physical therapy interventions.

Another issue related to confidence intervals and sample size calculation is whether one or multiple primary outcomes are nominated. Although adjustment for multiple primary outcomes is widely recommended and often used in clinical trials,18 it is more commonly used with p-values than confidence intervals.19 Therefore, it also seems worthwhile to observe whether physical therapy trials that nominate multiple primary outcomes adjust for multiple primary outcomes. Therefore, the primary aim of this study was to answer the following question:

• 1.

  How prevalent is the use of confidence intervals in the current reporting of between-group differences in randomised trials of physical therapy interventions?

  Secondarily we aimed to answer:

• 2.

  Where confidence intervals are used in the reporting of between-group differences, are the confidence intervals interpreted by the authors?

• 3.

  Do physical therapy trials achieve their intended sample size?

• 4.

  Where physical therapy trials nominate multiple primary outcomes, do they adjust for multiple primary outcomes?

The June 2022 update of PEDro contained 2832 trials published in 2021. About 2 % of trials were excluded because they were in-process or did not report a between-group comparison. None of the randomly sampled trials used Bayesian methods so none was excluded for this reason.

How prevalent is the use of confidence intervals?

The prevalence of confidence intervals for at least one outcome among the 100 trials published in 2021 was 47 % (95 % CI: 38, 57). This represents a marginal improvement over 2016, when 42 % (95 % CI: 29, 56) of trials reported some or all outcomes with confidence intervals (Fig. 1A). Among the 47 trials that presented confidence intervals, 36 trials (77 %) reported them for continuous outcomes (Fig. 1B). Among the 47 trials that presented confidence intervals, 45 trials (96 %) reported them numerically, none reported them graphically only, and only 2 (4 %) reported confidence intervals both numerically and graphically.

Fig. 1.

Percentage of trials from 2016 to 2021 that (A) reported 95 % confidence intervals for at least one outcome and (B) reported 95 % confidence intervals for continuous and/or dichotomous outcomes.

(0.22MB).

Trials that reported confidence intervals had a marginally higher PEDro Scale score than trials that did not report confidence intervals (Hodges-Lehmann median difference: 1; 95 % CI: 0, 1), as shown in Fig. 2A. Trials that reported confidence intervals had higher sample sizes than trials that did not report confidence intervals (Hodges-Lehmann median difference: 34; 95 % CI: 12, 66). (Fig. 2B).

Fig. 2.

Comparisons between the trials that did and did not report 95 % confidence intervals for two characteristics: (A) total PEDro score and (B) number of participants. Data are presented as median and IQR.

(0.08MB).

Overall, less than half of our sample of 100 trials reported confidence intervals. This represented an absolute increase of 5 % since 2016.9 The improvement observed over this 5-year period might be an indication of advances in awareness regarding the important role of confidence intervals in providing meaningful information about the precision of estimates of clinical interventions. These trends are a positive sign in reducing the entrenched habit of using p-values as the method of statistical inference.

Previous studies observed the prevalence of confidence intervals reporting ranging from 86 % of trials in epidemiology, 54 % in public health, 27 % in dermatology, and 9 % in biomedical research.17,23,24 The current reporting in physical therapy trials (47 %) was in the middle of this range. However, we did not restrict the inclusion of studies according to the publishing journals’ impact factor, as performed in previous studies. Instead, we included any journals indexed on PEDro, which comprehensively indexes physical therapy trials,20,21 thereby forming a representative sample for analysis. This might explain at least part of the differences in results across disciplines, as studies published in high impact journals are more likely to present confidence intervals.9,15

High impact journals tend to adhere to reporting guidelines, such as the Consolidated Standards of Reporting Trials (CONSORT), which require the presentation of estimated effect size and its precision.25 Several journals already endorse the use of CONSORT as a requirement for submission and there is a trend to increase the adoption of this policy across journals.26 This could be one of the factors related to the increase in reporting of confidence intervals observed between 2016 and 2021.

Confidence intervals are important to indicate the direction and size of the treatment effect, so improvements observed in the reporting of confidence intervals have important implications. Confidence intervals might help reduce “spin” (incorrect or misrepresentation of results) common in physical therapy research and help clinicians make more informed clinical decisions.27 To continuously improve the reporting of confidence intervals, editors, reviewers and researchers should be encouraged to require that clinical trialists report confidence intervals as a standard component of their results.7

Reporting of confidence intervals is critical, but it's only half the job. Confidence intervals must also be interpreted appropriately. To our knowledge, this is the first study that investigated the format of representation (numerical or graphical) and whether these estimates were interpreted by researchers. A sound interpretation by the original authors can help clinicians to understand confidence intervals correctly. Among the 100 trials, only 6 % (95 % CI: 3, 12) of studies both reported and interpreted a confidence interval for at least one outcome. Our results are consistent with previous literature where the interpretation of results in terms of clinical relevance is sparsely used across trials.17,28

Some clinicians may not have a strong background in statistics or may have a misunderstanding of confidence intervals. These factors can lead to challenges in interpreting the clinical implications of confidence intervals.29 Therefore, clinical trialists should discuss the clinical implications of the confidence intervals when they publish their results. This will help clinicians to appreciate the clinical relevance of the study findings. Providing an interpretation should involve discussion of the clinical implications of the range of values within the confidence interval. This might include reference to the null value being inside or outside the confidence interval, and reference to location of the confidence interval relative to the smallest worthwhile effect or a clinically relevant threshold.

Another aspect that could facilitate interpretation of confidence intervals by clinicians could be the format of presentation. Among the 47 trials that included confidence intervals, 96 % presented a numerical representation. Previous studies demonstrated that format of presentation of patient-reported outcomes in clinical trials may be associated with how accurately they are interpreted by clinicians.30–32 Visual presentation strategies may be further explored, to improve efficacy in data communication around confidence intervals and proposed as an alternate method of presenting findings. Graphic presentations can provide an intuitive understanding regarding an outcome's estimates and help reduce the challenge for non-statisticians to understand the clinical implications of confidence intervals.

The sample size of a clinical trial directly affects confidence intervals. Lower sample sizes reduce the likelihood that the study generates estimates that are representative of the true effect in the wider patient population from which the study cohort was sampled. To acknowledge this larger degree of uncertainty in the findings, smaller samples generate wider confidence intervals. When a study's estimate is accompanied by a wide confidence interval, clinicians should acknowledge that uncertainty about what the true average effect of the intervention is. They need to be prepared to explain to patients that the evidence hasn't yet precisely estimated the average effect of the intervention on that outcome. They should be prepared to explain that the average effect might be favorable at the upper limit of the confidence interval, unfavorable at the lower limit of the confidence interval, or anywhere in between.

Our results showed some interesting findings regarding sample size. First, about one-third of the trials did not report a sample size calculation. Although some issues have been raised regarding common approaches to sample size calculation in clinical trials,10 it is disappointing that this proportion of clinical trialists did not consider sample size in any formal way when planning their study. Second, among 68 trials that did report a sample size calculation, 59 achieved or exceeded the required sample size. This suggests that calculating a target sample size might inspire researchers to ensure it is achieved. Overall, though, there is still plenty of potential to improve the calculation and achievement of the required sample size, given that only 59 % (95 % CI 49, 68) achieved both of these elements. This problem has been observed elsewhere. For example, Gianola et al33 identified that only 40 % of randomised controlled trials that analysed rehabilitation interventions for low back pain were adequately powered.

The last aim of our study was to determine the prevalence of adjustment for multiplicity among physical therapy trials that reported multiple primary outcomes. Among the 30 trials that nominated multiple primary outcomes, 11 (37 %) made an adjustment for multiplicity. Therefore, among the full cohort of 100 trials, only 19 % (95 % CI: 13, 28) had a problem with unadjusted multiplicity. Although it affected a relatively small proportion of the trials, adjustments for multiplicity are required to reduce the possibility of type I error.18 Failure to adjust for multiple comparisons in randomised trials can result in unreliable findings, which can have negative consequences for patient care. Although adjustments for multiple comparisons are important, they also reduce the statistical power of the analysis for a given sample size and therefore widen the confidence intervals.18 Adjustment of the sample size to account for this is recommended.

The strengths in our design included the assessment of multiple trial characteristics with reliable data extraction (two independent reviewers). Additionally, we used random selection to generate a large representative sample of trials, improving the generalisability of the findings. Our study presents important implications because the findings indicate the need for greater attention to the reporting of confidence intervals in randomised controlled trials. One limitation is that, despite applying no language restrictions during random sampling, only English-language trials were selected. This is likely because only 3 % of the eligible trials from which the 100 trials were selected were published in languages other than English. Another limitation is that we did not assess whether the studies that adjusted for multiple comparisons in their analysis also adjusted their sample size accordingly. This could be the focus of future research. Additionally, further research could address whether confidence intervals are calculated correctly and interpreted appropriately.

Around half of trials of physical therapy interventions published in 2021 reported a confidence interval around at least one between-group difference. This represents an increase of 5 % from 5 years earlier. Very few trials interpreted the confidence interval and presented the confidence interval graphically. Most trials of physical therapy interventions report a sample size calculation, and achieve the required sample size. There is still a need to increase the nomination of a primary outcome and, if multiple primary outcomes are nominated, the use of adjustment for multiple comparisons.