A multicenter mixed-methods study was performed, in which 22 indicators of quality of care in hospital-based physical therapy –previously selected from a systematic review8– were sequentially tested and refined. The design of this multi-stepped mixed-methods study was observational, descriptive, and cross-sectional in each study site. The feasibility of the 9 management indicators was tested based on the availability of existing data sources and the time used to measure a single case. Whilst we evaluated the 13 clinical indicators for their measurability (feasibility) and their inter-rater agreement (i.e., their reliability), following the GRRAS guidelines.9

Three hospitals were selected by convenience recruitment criteria, which were based on: a) the availability and willingness of the heads of physical therapy departments to collaborate; b) their location in three different regions of public hospitals of Spain within the Spanish National Health System (SNHS), with decentralized and autonomously managed governance and provision of health services. Consequently, they differed in aspects such as data registration systems, the organization of rehabilitation and physical therapy units, and the functional role of physical therapy supervisors.

The study was approved by the ethics committees of Hospital Universitario Reina Sofía (Ref. 313/10-11-21), Hospital Universitario Fundación Alcorcón (Ref. 21/162), and Hospital El Bierzo (Ref. 21155). In addition, these approvals were ratified by the Ethics Committee of the Universidad Católica San Antonio de Murcia (UCAM).

The sample size for clinical indicators, which measure healthcare processes, was determined using the following formula:10 n=(z2×p×(1−p))i2 where z = 1.96, i is the precision interval (0.15) and p is the expected proportion of compliance (0.5)10 The sample size was estimated at 43 observations per indicator. We randomly selected 60 cases (allowing oversampling in the event of substitution) from each study population, as presented in the indicator reference sheets (Appendix A).

Unit management indicators were intended to be measured directly from readily exploitable or accessible records to the physical therapy supervisor. No sampling was required.

Patients were identified using ICD-10 codes recorded in the Spanish Minimum Basic Data Set (MBDS), according to each indicator's requirements, including both inpatients and outpatients. Systematic random sampling was used.

To assess and measure the performance (quality level) of the 13 clinical indicators, clinical records (electronic or paper medical records) were used, along with additional auxiliary records. This data collection was performed by two independent, non-unique external evaluators to the hospitals, using standardized ad hoc formats: an Excel spreadsheet for clinical indicators (Appendix B) and a Google Form for management indicators (Appendix C). Evaluators were asked to keep a personal log, in the form of a journal, to record the difficulties and concerns they experienced, as well as to identify the optimal documents and information routes.

During testing, indicator wording and assessment criteria were iteratively refined. When possible, optimal data flows and adjusted data collection tools were implemented to address collection challenges and inter-rater discrepancies. These modifications were applied post-evaluation to preserve evaluator independence.

Both feasibility and inter-rater reliability were quantitatively and qualitatively. Feasibility was categorized into five groups: 1a. High (Available in all hospitals and without high cost); 1b. Moderate (Available in all hospitals but requiring high cost); 2a. Low (Available in at least one hospital, without high cost); 2b. Very low (Available in at least one hospital, but requiring high cost); 3. Not feasible/not measurable.

Evaluators took personal notes in a diary. Any situation that prevented or hindered the assessment of indicators during data extraction was recorded, and the estimated time required to measure one case for each indicator was also recorded. High cost occurred when the data needed to be searched across more than 3 documents, the language was not standardized, or the time per case exceeded 30 minutes. At the end of the data collection, evaluators discussed the difficulties observed and concerns experienced, classified them, and described them.

Inter-rater reliability was tested by means of the percentage of observed agreement between the two evaluators (NI and BA-C) and the calculation of the inter-observer kappa index (κ) for each indicator measured in clinical records. Since κ values can be severely affected by the observed prevalence of the classification categories and assessment biases, the Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) was calculated. Both indices were interpreted using the criteria of Landis and Koch described in 1977,11 allowing decisions to be made based on the results for each study site. Inter-rater reliability was dichotomized (PABAK ≥ 0.7) to determine which hospitals were reliable and which were not. Finally, to identify areas of opportunity, the quality level (compliance and 95% confidence interval) of each indicator and sub-indicator was estimated. All calculations were performed in Excel spreadsheets.

Of the 22 indicators, 17 were feasible in at least two hospitals and 13 in all hospitals (Table 2). Hospital 1 had the greatest difficulty measuring the indicators (feasibility problems).

The main difficulties identified by the evaluators can be summarized as follows: inadequate identification of the pathologies of interest due to ICD codes lacking specificity; limited identification of the initial diagnosis or index episode in patients with chronic conditions; and failure to record essential evaluation information. Detailed qualitative information for each indicator and sub-indicator is provided in Appendix D. The clinical indicators that were not feasible in any hospital were indicators 1.2 and 1.3 on achieving treatment goals and the percentage of follow-up visits once achieved, and 3.3 on patients newly diagnosed with rheumatoid arthritis. Among the management indicators, the two non-feasible indicators were 6.3, sessions lost due to transport, and 6.8, readmission within 90 days for patients referred to outsourced care or non-public services.

Regarding the three indicators of therapeutic objectives, following indicator 1.1, there was practically no objective recorded. Meanwhile, as with the other indicators focusing on respiratory diseases (4.1 and 4.2), the indicator for rheumatoid arthritis was hampered by difficulties in identifying the onset of a diagnosis or episode and by unspecified ICD codes. In the case of the two non-feasible management indicators, the lack of recorded data needed to collect or calculate them is worth noting.

Finally, after assessing each study location, major adjustments were made to the non-feasible indicators. Most notable was the reformulation of indicator 6.8 (Rate of patients referred with readmission within 90 days due to unresolved problem), now simplified to ‘Rate of patients referred to private healthcare’ and targeted at specific population groups.

Hospital 1 was the site where the fewest indicators could be measured and where the greatest differences in evaluator agreement were observed. This was progressively reduced in the following locations. There was only one criterion with a PABAK of 0.0 (‘at least one therapeutic objective’ criterion) due to disagreements regarding what constituted a complete therapeutic objective and the diversity of formats to collect that information. The remaining indicators and most of the other criteria (sub-indicators) achieved acceptable PABAK values (median 0.95, interquartile range 0.91-1). All the observed concordance, kappa, and PABAK values are available in Appendix E, while the qualitative analysis of the discordance between evaluators is summarised in Appendix F.

Table 3 describes the quality level of the indicators and their measurable sub-indicators. H1 had the lowest compliance levels. The indicators that were met by more than 75% were indicator No. 1.1 (86.05% ± 10.36%) and indicator No. 2.2 (79.55% ± 11.92%) in H2, with relatively narrow 95% CIs indicating precise estimates of compliance. There were four indicators below 5% in H1 (2.2, 3.1.A, 4.1, 4.2), four in H2 (1.1, 2.1, 3.2, 4.1), and five in H3 (2.1, 3.2, 4.1, 4.2, 5.1). Table 3 shows the differences in the management indicators results across the 3 hospitals.

Two groups of indicators are established from our results. The first group consists of 13 indicators (9 clinical and 4-unit management indicators) that have been shown to be measurable and reliable in this study (Table 4). The second includes the 9 indicators that could not be proven sufficiently reliable or measured, at least not without incurring high measurement costs, during the study (Table 5). Data collection tools and QI were refined as each hospital was surveyed. Final indicator sheets and final data collection tools are available in Appendices A, B, and C, respectively.

Our indicators enable the assessment of coordination and resources in hospital physical therapy units, supporting internal analysis and inter-hospital comparison to foster organizational improvement.

This study tested 17 indicators across three public hospitals within the SNHS. These indicators can be immediately incorporated into a dashboard to improve the management of physical therapy units,39 covering both administrative and clinical aspects. Further studies might narrow the focus to assess the detected needs of local physical therapy units more comprehensively and specifically.40,41 Using these indicators, physical therapists can enhance their clinical care and adhere to current facility-level healthcare decision-making and management. There remains a substantial opportunity for physical therapy units to demonstrate the added value of their work and contributions to both other health disciplines42 and the general population.43

The selected indicators may be complemented by context-specific or population-focused measures, including structural or outcome indicators (e.g., research activity, protocol availability, or relevant health outcomes), aligning with the RECAL project evaluations of the SNHS.44

Our study has some limitations. Firstly, we acknowledge that our feasibility results are context-dependent and primarily depend on the availability of high-quality, well-structured information and the capabilities to exploit it. There is no guarantee that the indicators we identified as feasible are measurable in every hospital. The study locations may not be representative of the characteristics of every Spanish public hospital. Although the findings may not represent all Spanish public hospitals, the study included hospitals from different administrative regions, reflecting diverse health information systems and data sources, thereby increasing the generalizability of the results. This diversity was considered in the assessment of feasibility and inter-rater reliability to adapt collection tools to different contexts and a wider range of evaluation circumstances.

Nevertheless, variability in clinical documentation practices and registry structures across Spain may still limit the generalizability of the results. Secondly, the estimated quality levels are partly influenced by inter-rater agreement in applying certain indicators; limited agreement may increase measurement error and, consequently, affect the resulting estimates. Although no precise estimate can be made of the performance of a broader regional or national perspective, such perspectives serve as a baseline for these indicators, which have not previously been measured in Spanish hospitals. In addition, this set of indicators has been shown to identify opportunities for improvement in the hospitals studied, supporting their potential usefulness for other hospitals.

It is important to acknowledge the potential for evaluator bias, including a degree of subjectivity. For process indicators, the evaluators were external to the hospitals. Therefore, any potential bias may be more related to the evaluators’ own experience in retrieving the necessary data to measure the indicators. This aspect, however, can also be considered a strength, as it reflects the practical challenges of accessing the information required to assess service quality.

In contrast, for the management indicators, the data were provided by the heads of physical therapy departments, who are the end users of these indicators. The issue of whether quality assessment should be conducted internally or externally has long been debated.45,46,47 In this study, our main concern was not to externally judge the quality of each hospital, but rather to determine whether the necessary data to assess that quality could be obtained. Therefore, although the data on the management indicators may have been influenced by a certain subjectivity on the part of the supervisors of physical therapy, we believe that the results obtained reinforce the idea that the usefulness of any indicator lies not only in its conceptual validity, but also in the feasibility of its routine application by healthcare professionals within their own organizational environments.

Although the results of this study confirm the feasibility of using certain indicators, we cannot affirm their widespread use or applicability in all cases, and it is necessary to evaluate the implementation of adaptations tailored to particularities.