Potentially eligible SRs (with or without meta-analysis) should have been published between 2011 and 2020 as full-text scientific articles in any of the 68 rehabilitation journals indexed under the “rehabilitation” category in InCites Journal Citation Report. A standard or consensus definition of a SR does not exist.18 Therefore we considered a review to be systematic if it displayed the following characteristics19:

• •

  A clearly defined research question

• •

  Declaration of the sources that were searched

• •

  Inclusion and exclusion criteria and selection methods

• •

  Critical appraisal and reporting of the quality/ROB of the included studies

• •

  Information about data analysis and synthesis

Narrative reviews, mixed-methods reviews, qualitative evidence synthesis, umbrella reviews, scoping reviews, editorials, letters, and news reports were excluded.

Journal tags for the journals were identified in Medline, and a detailed search strategy was created to find all SRs published from 2011 to 2020 in this database (see online supplementary material Table S.2). The search was performed only in Medline, as all rehabilitation journals are indexed in this database. Titles, abstracts, and full texts of all articles retrieved were screened for eligibility in a double-blinded process by two independent reviewers who selected potential articles based on study selection criteria. A third reviewer resolved any disagreement. After this process, 200 reports (our study sample) were randomly selected by an independent author using a free web service (Research Randomizer - www.randomizer.org) to create the random sequence list. Randomisation was stratified by publication date and journal ranking (quartile range; Q1, Q2, Q3, Q4 – based on InCites15) to include an equal number of studies from 2011 to 2015 (n=25 for each quartile range) and from 2016 to 2020 (n=25 for each quartile range).

Full texts were stored in EndNote X7 (Thomas Reuters, Philadelphia, Pennsylvania, USA). A data extraction form was used, and the following characteristics were extracted: first author; publication year; journal characteristics (open access vs hybrid); country; rehabilitation field; protocol registration (yes/no); completeness of reporting (see below); ROB with the ROBIS tool (see below). Two reviewers performed data extraction and the ROB assessment for each SR, each one independently; when necessary, any disagreement was resolved by a third reviewer. For the publication option (open access or hybrid) we checked if journals had changed their policy over time (by checking the journal website or asking the editor directly).

Completeness of reporting was calculated as adherence to the 27-item PRISMA 2009 checklist4 (from here onwards indicated as PRISMA checklist). Discrepancies between authors on the application of checklist items were resolved through consensus by referring to the published explanation of the PRISMA checklist.4 Inter-rater agreement was evaluated for 30 SRs (15% of the entire sample) which were scored independently by two authors with post-graduate training in clinical epidemiology and critical appraisal. This method was chosen to ensure reproducibility and to align the appraisers. However, PRISMA checklist is not an assessment checklist, so it does not need to be performed independently by two authors.

Following the explanation and elaboration statements of the PRISMA checklist, each item was marked with “1” if it was well described, with “0” if incomplete or missing, and with “NA” if not applicable. An item was defined as “NA” when the authors did not describe it, and if it was clear (by reading the protocol and/or the full text) that such information was justifiably missing (e.g. if authors did not perform any meta-analyses, item 23 was not applicable). Adherence to each item and overall adherence to the PRISMA checklist for each study were calculated (from 0% to 100% with 0 representing zero adherence and 100% full adherence), weighting against the number of applicable items. As the aim of this study was to investigate completeness of reporting, the authors of the studies included were not contacted for information omitted from manuscripts.

The ROBIS tool11 was used to assess the ROB in the studies included. For each SR included, a ROB judgment was assigned for each of the four domains (1. the specification of study; 2. methods used to identify and/or select studies eligibility criteria; 3. methods used to collect data and appraise studies; 4. synthesis and findings). Possible assessment was ‘‘high,’’ ‘‘low,’’ or ‘‘some concerns.’’ An overall ROB score for each study was assigned as suggested by the instructions of the tool. The results were graphically summarised through the ROB graph obtained with the ROBVIS Tool.20

The primary analysis addressed:

• •

  The completeness of reporting in each study (see above) calculated as the overall adherence to the PRISMA checklist. It was estimated (in percentage) as the total number of items described and reported out of the total number of applicable items.

• •

  The completeness of reporting for each item and each section of the PRISMA checklist in all studies, calculated as (for each item; in percentage) the number of times that one item is described and reported out of the total number of studies in which the item could potentially appear in.

• •

  The relationship between the completeness of reporting in each study (see above) and studies with low or high ROB. We hypothesised that studies with domains with a high ROB display poorer reporting than those with a low ROB. The relationship was investigated through descriptive statistics and linear regression analyses (both univariable and multivariable) between the overall adherence to the PRISMA checklist as the dependent variable and the ROB (for each ROBIS domain and overall ROB) as independent variables.

Secondary analysis:

• •

  Linear regression analyses (both univariable and multivariable) were performed to assess the association between the completeness of reporting (calculated as the overall adherence to the PRISMA checklist – see above) as the dependent variable and the following characteristics as independent variables: publication year; journal ranking (quartile range: Q1, Q2, Q3, Q4); publication options (open access vs hybrid); study protocol prospective registration (no vs yes).

For each linear regression model, the assumptions of linearity, homoscedasticity, independence, and normality were checked. For all regression analyses, these assumptions were met.

The mean overall adherence among our sample was 61.4%, ranging from 100% to 16% (See online supplementary material Figure S.1). Overall, the mean adherence varied across items (Table 1). The most frequently reported item was #1 in 99.5% of cases; the least reported one was the ROB across studies (#22) in 12% of cases. Some items were not applicable in most SRs (Table 1), which may have influenced the calculation of adherence in these cases (e.g. item #16 concerning the description of additional analyses, where adherence was 35% and NA=120). In general, the highest adherence rates were in the title item/section (99.5%) and discussion section (83%), the lowest in the funding section (in which there is only item #27) (39%) (Fig. 2).

Fig. 2.

Adherence across PRISMA sections and overall adherence.

(0.19MB).

Overall, studies with high ROB showed poorer reporting than those with low ROB (Table 2). The results of ROBIS analysis are reported in supplementary material Table S.5. The descriptive analysis showed that reporting of the methods section presented the largest differences between studies at high and low ROB in all ROBIS domains (Table 2).

The univariable linear regression analysis showed a negative association between high ROB and completeness of the reporting (adherence to the PRISMA checklist) in all domains (see supplementary material Table S.6). Still, when we entered all independent variables in the same model, the multivariable linear regression analyses (Table 3 – primary analysis) reported an association between completeness of reporting and ROB in domains 1, 2, and overall ROB.

Having a high ROB in domain 1 (B= -5.7%; 95%CI -10.1, -1.3) and domain 2 (B= -5.4%; 95%CI -9.7, -1.1) are significant predictors of lower overall adherence to the PRISMA checklist. If we look at the overall ROB, having a high overall ROB is a significant predictor of lower adherence (B=-7.1%; 95%CI -12.1, -2.0).

Univariable linear regression models showed a significant relationship between journal IF, publication year, study protocol registration, and completeness of the reporting (see supplementary material Table S.7). When we entered all independent variables in the multivariable linear regression model, only journal IF and protocol registration were significant predictors (see Table 3 – secondary analysis). Studies published in fourth quartile journals displayed a lower overall adherence (B= -7.2%; 95%CI -13.2, -1.3) than those published in first quartile journals; the overall adherence increased (B= 11.9%; 95%CI 5.9, 18.0) if the SR protocol was registered.

We used the PRISMA checklist to assess the completeness of reporting, despite the checklist being a guide for writing and not for evaluating quality. We are aware that despite being consistent with previous meta-research studies,12,13,22 this may represent a limitation.32 To overcome this issue, we have tried to define how we have reached the scores clearly, and we ran a preliminary pilot test to increase inter-assessor agreement, especially on items requiring interpretation (sometimes indicated by phrases like “if relevant” or “if applicable”). Moreover, between 2011 and 2020 many other SRs have been published and our sample (200 hits) may not be representative of all studies published in such time frame. However, our choice is based on the sample size used in other similar meta-research studies12-14,33 and we believe it is representative of the characteristic of the SRs published in the rehabilitation journals.