Journal Information
Vol. 26. Issue 5.
(01 September 2022)
Share
Share
Download PDF
More article options
Visits
...
Vol. 26. Issue 5.
(01 September 2022)
Systematic Review
Open Access
Adherence to the PRISMA statement and its association with risk of bias in systematic reviews published in rehabilitation journals: A meta-research study
Visits
...
Tiziano Innocentia,b,
Corresponding author
t.innocenti@vu.nl

Corresponding author at: De Boelelaan 1085, 1081 HV Amsterdam, the Netherlands.
, Daniel Fellerc, Silvia Giagiod,e, Stefano Salviolib,f, Silvia Minnuccig, Fabrizio Brindisinog,h, Carola Cosentinof, Leonardo Pianoi, Alessandro Chiarottoa,j, Raymond Osteloa,k
a Department of Health Sciences, Faculty of Science, Vrije Universiteit Amsterdam, Amsterdam Movement Sciences, the Netherlands
b GIMBE Foundation, Bologna, Italy
c Provincial Agency for Health of the Autonomous Province of Trento, Trento, Italy
d Division of Occupational Medicine, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
e Department of Biomedical and Neuromotor Sciences (DIBINEM), Alma Mater Studiorum, University of Bologna, Bologna, Italy
f Department of Neuroscience, Rehabilitation, Ophthalmology, Genetics, Maternal and Child Health, University of Genoa, Genoa, Italy
g Department of Clinical Science and Translational Medicine, University of Rome Tor Vergata, Rome, Italy
h Department of Medicine and Health Science “Vincenzo Tiberio”, University of Molise, Campobasso, Italy
i Unit of Rehabilitation and Functional Recovery, Fondazione dei Santi Lorenzo e Teobaldo, Rodello, Italy
j Department of General Practice, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
k Department of Epidemiology and Data Science, Amsterdam UMC, Location VUmc, Amsterdam Movement Sciences research institute, the Netherlands
Highlights

  • The mean overall adherence to the PRISMA checklist across the sample of systematic reviews published in rehabilitation journals was 61.4%.

  • A high overall risk of bias was a significant predictor of lower adherence (B=−7.1%; 95%CI −12.1, −2.0).

  • Studies published in fourth quartile journals displayed a lower overall adherence than those published in the first quartile.

  • No association between adherence and publication options and publication year was found.

  • The overall adherence increased (B= 11.9%; 95%CI 5.9, 18.0) when the SR protocol was registered

Article information
Abstract
Full Text
Bibliography
Download PDF
Statistics
Figures (2)
Additional material (1)
Abstract
Background

Systematic reviews (SRs) and meta-analyses are essential resources for the clinicians. They allow to evaluate the strengths and the weaknesses of the evidence to support clinical decision-making if they are adequately reported. Little is known in the rehabilitation field about the completeness of reporting of SRs and its relationship with the risk of bias (ROB).

Objectives

Primary: 1) To evaluate the completeness of reporting of systematic reviews (SRs) published in rehabilitation journals by evaluating their adherence to the PRISMA 2009 checklist, 2) To investigate the relationship between ROB and completeness of reporting. Secondary: To study the association between completeness of reporting and journals and study characteristics.

Methods

A random sample of 200 SRs published between 2011 and 2020 in 68 rehabilitation journals was indexed under the “rehabilitation” category in the InCites database. Two independent reviewers evaluated adherence to the PRISMA checklist and assessed ROB using the ROBIS tool. Overall adherence and adherence to each PRISMA item and section were calculated. Regression analyses investigated the association between completeness of reporting, ROB, and other characteristics (impact factor, publication options, publication year, and study protocol registration).

Results

The mean overall PRISMA adherence across the 200 studies considered was 61.4%. Regression analyses show that having a high overall ROB is a significant predictor of lower adherence (B=-7.1%; 95%CI -12.1, -2.0). Studies published in fourth quartile journals displayed a lower overall adherence (B= -7.2%; 95%CI -13.2, -1.3) than those published in first quartile journals; the overall adherence increased (B= 11.9%; 95%CI 5.9, 18.0) if the SR protocol was registered. No association between adherence, publication options, and publication year was found.

Conclusion

Reporting completeness in rehabilitation SRs is suboptimal and is associated with ROB, impact factor, and study registration. Authors of SRs should improve adherence to the PRISMA guideline, and journal editors should implement strategies to optimize the completeness of reporting.

Keywords:
Meta-research
Randomised controlled trial
Rehabilitation
Reporting guidelines
Research quality
Risk of bias
Full Text
Introduction

Systematic reviews (SRs) and meta-analyses provide the highest quality of scientific evidence and are essential resources shaping the clinical decision-making process.1 They represent an efficient way for clinicians to keep up to date with the current evidence and provide a starting point for developing clinical guidelines.2,3 Therefore, adequate reporting of SR methods and results is essential for evaluating the strengths and weaknesses of the evidence provided.4 Moreover, complete and transparently reported research aids reproducibility and critical appraisal.5 Indeed, critical appraisal of study methods enables assessment of the extent to which results and author interpretations overestimate or underestimate study effects and highly depend on what is reported in research articles. For example, if a study has better reporting of all necessary information, this could positively impact the transparency of the risk of bias (ROB) evaluation.6

Problems in the reporting of a scientific study can affect research in different ways. For example, it is known that study methods are frequently not described in adequate detail and that results are presented ambiguously, incompletely, or selectively.7 The consequence is that many reports cannot be used for replication purposes, are potentially a waste of resource, or could even be harmful.8 Reporting guidelines (RGs) have been developed to help authors optimally report key aspects of their research in their manuscripts.

The lack of standardisation and poor quality of reporting among SRs and meta-analyses led to the development of the Quality of Reporting of Meta-analyses (QUOROM) statement in 1999.9 In 2009, an updated version of the QUORUM, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement was launched4: it is a consensus-based minimum set of items to be reported in SRs with or without meta-analyses. It consists of 27 items that should be reported in the title, abstract, methods, results, and discussion of a SR and meta-analysis, including source of funding. The PRISMA statement was updated in 2020.10

The ROB in systematic reviews (ROBIS) tool, specifically designed to assess ROB in SRs was published in 2016.11 Following a domain-based approach, ROBIS covers the evaluation of the internal validity of the review process and the relevance of the review question for its users. It comprises four specific domains (eligibility criteria, identification and selection of studies, data collection and study appraisal, synthesis and findings) and results in ROB judgment (both on each domain and overall) which can be ‘‘high,’’ ‘‘low,’’ or ‘‘unclear.’’

Previous studies have assessed the adherence to the PRISMA 2009 checklist in published SRs in the medical field12; one study13 in the field of orthopaedic surgery analysed the quality of reporting of relevant studies which were published in the top five highest impact factor (IF) orthopaedic journals and found that only 68% of items of the PRISMA 2009 statement were reported. Little is known about reporting in the rehabilitation field: a previous study14 confirmed that most of the SRs authors (∼ 65%) publishing in high impact rehabilitation journals did not mention the use of PRISMA, and approximately 40% of those who declared using PRISMA did not do so appropriately.

To our knowledge, completeness of reporting and the relationship between ROB and completeness of reporting for SRs published in rehabilitation journals have not been systematically evaluated.

The primary objectives of this meta-research study were: 1) to evaluate the completeness of reporting in SRs published in rehabilitation journals through the evaluation of the adherence to the PRISMA checklist4; 2) to investigate the relationship between the completeness of reporting and the ROB assessed with the ROBIS tool. The secondary objective was to study the association between the completeness of the reporting and the characteristics of the SRs and journals, such as impact factor, publication medium, publication year, and protocol registration. We believe that a better understanding of these topics is important to help researchers and readers in producing and assessing the evidence in a more accurate manner.

Methods

A cross-sectional analysis was conducted for a random sample of 200 SRs published between 2011 and 2020 in all 68 journals indexed under the “rehabilitation” category in InCites Journal Citation Report.15 The 68 rehabilitation journals indexed in InCites and their characteristics are reported in the online supplementary material (see Table S.1). The project protocol was publicly posted before data extraction (prospectively registered) on the medRxiv preprint server.16 Given that a specific reporting checklist for meta-research studies is currently under development,17 we followed the PRISMA 2020 checklist10 in the reporting of this manuscript.

Study selection criteria

Potentially eligible SRs (with or without meta-analysis) should have been published between 2011 and 2020 as full-text scientific articles in any of the 68 rehabilitation journals indexed under the “rehabilitation” category in InCites Journal Citation Report. A standard or consensus definition of a SR does not exist.18 Therefore we considered a review to be systematic if it displayed the following characteristics19:

  • A clearly defined research question

  • Declaration of the sources that were searched

  • Inclusion and exclusion criteria and selection methods

  • Critical appraisal and reporting of the quality/ROB of the included studies

  • Information about data analysis and synthesis

Narrative reviews, mixed-methods reviews, qualitative evidence synthesis, umbrella reviews, scoping reviews, editorials, letters, and news reports were excluded.

Study selection process

Journal tags for the journals were identified in Medline, and a detailed search strategy was created to find all SRs published from 2011 to 2020 in this database (see online supplementary material Table S.2). The search was performed only in Medline, as all rehabilitation journals are indexed in this database. Titles, abstracts, and full texts of all articles retrieved were screened for eligibility in a double-blinded process by two independent reviewers who selected potential articles based on study selection criteria. A third reviewer resolved any disagreement. After this process, 200 reports (our study sample) were randomly selected by an independent author using a free web service (Research Randomizer - www.randomizer.org) to create the random sequence list. Randomisation was stratified by publication date and journal ranking (quartile range; Q1, Q2, Q3, Q4 – based on InCites15) to include an equal number of studies from 2011 to 2015 (n=25 for each quartile range) and from 2016 to 2020 (n=25 for each quartile range).

Data extraction

Full texts were stored in EndNote X7 (Thomas Reuters, Philadelphia, Pennsylvania, USA). A data extraction form was used, and the following characteristics were extracted: first author; publication year; journal characteristics (open access vs hybrid); country; rehabilitation field; protocol registration (yes/no); completeness of reporting (see below); ROB with the ROBIS tool (see below). Two reviewers performed data extraction and the ROB assessment for each SR, each one independently; when necessary, any disagreement was resolved by a third reviewer. For the publication option (open access or hybrid) we checked if journals had changed their policy over time (by checking the journal website or asking the editor directly).

Assessing the completeness of reporting

Completeness of reporting was calculated as adherence to the 27-item PRISMA 2009 checklist4 (from here onwards indicated as PRISMA checklist). Discrepancies between authors on the application of checklist items were resolved through consensus by referring to the published explanation of the PRISMA checklist.4 Inter-rater agreement was evaluated for 30 SRs (15% of the entire sample) which were scored independently by two authors with post-graduate training in clinical epidemiology and critical appraisal. This method was chosen to ensure reproducibility and to align the appraisers. However, PRISMA checklist is not an assessment checklist, so it does not need to be performed independently by two authors.

Following the explanation and elaboration statements of the PRISMA checklist, each item was marked with “1” if it was well described, with “0” if incomplete or missing, and with “NA” if not applicable. An item was defined as “NA” when the authors did not describe it, and if it was clear (by reading the protocol and/or the full text) that such information was justifiably missing (e.g. if authors did not perform any meta-analyses, item 23 was not applicable). Adherence to each item and overall adherence to the PRISMA checklist for each study were calculated (from 0% to 100% with 0 representing zero adherence and 100% full adherence), weighting against the number of applicable items. As the aim of this study was to investigate completeness of reporting, the authors of the studies included were not contacted for information omitted from manuscripts.

Assessing the risk of bias

The ROBIS tool11 was used to assess the ROB in the studies included. For each SR included, a ROB judgment was assigned for each of the four domains (1. the specification of study; 2. methods used to identify and/or select studies eligibility criteria; 3. methods used to collect data and appraise studies; 4. synthesis and findings). Possible assessment was ‘‘high,’’ ‘‘low,’’ or ‘‘some concerns.’’ An overall ROB score for each study was assigned as suggested by the instructions of the tool. The results were graphically summarised through the ROB graph obtained with the ROBVIS Tool.20

Data analysis

The primary analysis addressed:

  • The completeness of reporting in each study (see above) calculated as the overall adherence to the PRISMA checklist. It was estimated (in percentage) as the total number of items described and reported out of the total number of applicable items.

  • The completeness of reporting for each item and each section of the PRISMA checklist in all studies, calculated as (for each item; in percentage) the number of times that one item is described and reported out of the total number of studies in which the item could potentially appear in.

  • The relationship between the completeness of reporting in each study (see above) and studies with low or high ROB. We hypothesised that studies with domains with a high ROB display poorer reporting than those with a low ROB. The relationship was investigated through descriptive statistics and linear regression analyses (both univariable and multivariable) between the overall adherence to the PRISMA checklist as the dependent variable and the ROB (for each ROBIS domain and overall ROB) as independent variables.

Secondary analysis:

  • Linear regression analyses (both univariable and multivariable) were performed to assess the association between the completeness of reporting (calculated as the overall adherence to the PRISMA checklist – see above) as the dependent variable and the following characteristics as independent variables: publication year; journal ranking (quartile range: Q1, Q2, Q3, Q4); publication options (open access vs hybrid); study protocol prospective registration (no vs yes).

For each linear regression model, the assumptions of linearity, homoscedasticity, independence, and normality were checked. For all regression analyses, these assumptions were met.

Results

The study selection process is summarised in Fig. 1. Online supplementary material includes characteristics of included systematic reviews (Table S.3) and their complete references (Table S.4).

Fig. 1.

Flowchart of the study selection process.

(0.21MB).
Completeness of reporting (adherence to the PRISMA checklist)

The mean overall adherence among our sample was 61.4%, ranging from 100% to 16% (See online supplementary material Figure S.1). Overall, the mean adherence varied across items (Table 1). The most frequently reported item was #1 in 99.5% of cases; the least reported one was the ROB across studies (#22) in 12% of cases. Some items were not applicable in most SRs (Table 1), which may have influenced the calculation of adherence in these cases (e.g. item #16 concerning the description of additional analyses, where adherence was 35% and NA=120). In general, the highest adherence rates were in the title item/section (99.5%) and discussion section (83%), the lowest in the funding section (in which there is only item #27) (39%) (Fig. 2).

Table 1.

Mean adherence across each item of the PRISMA 2009 Checklist in systematic reviews published in rehabilitation journals (n = 200).

PRISMA 2009 ItemsStudies where item was not applicable (n)  Mean adherence (%) calculated in studies where item was applicable 
Title   
  Identify the report as a systematic review, meta-analysis, or both.  99.5 
Abstract   
Structured summary  Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number.  64.0 
Introduction         
Rationale  Describe the rationale for the review in the context of what is already known.  97.5 
Objectives  Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS).  52.5 
Methods   
Protocol and registration  Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number.  12.5 
Eligibility criteria  Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale.  71.5 
Information sources  Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.  61.0 
Search  Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.  49.0 
Study selection  State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis).  75.0 
Data collection process  10  Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators.  44.5 
Data items  11  List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made.  59.0 
Risk of bias in individual studies  12  Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis.  77.0 
Summary measures  13  State the principal summary measures (e.g., risk ratio, difference in means).  21  52.5 
Synthesis of results  14  Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each meta-analysis.  23  48.0 
Risk of bias across studies  15  Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).  17.5 
Additional analyses  16  Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.  120  35.0 
Results   
Study selection  17  Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.  83.5 
Study characteristics  18  For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.  94.5 
Risk of bias within studies  19  Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).  85.0 
Results of individual studies  20  For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.  57.0 
Synthesis of results  21  Present results of each meta-analysis done, including confidence intervals and measures of consistency.  54.5 
Risk of bias across studies  22  Present results of any assessment of risk of bias across studies (see Item 15).  12.5 
Additional analysis  23  Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]).  120  37.0 
Discussion   
Summary of evidence  24  Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).  83.0 
Limitations  25  Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias).  77.0 
Conclusions  26  Provide a general interpretation of the results in the context of other evidence, and implications for future research.  89.0 
Funding   
Funding  27  Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.  39.0 
Fig. 2.

Adherence across PRISMA sections and overall adherence.

(0.19MB).
Relationship between risk of bias and completeness of reporting

Overall, studies with high ROB showed poorer reporting than those with low ROB (Table 2). The results of ROBIS analysis are reported in supplementary material Table S.5. The descriptive analysis showed that reporting of the methods section presented the largest differences between studies at high and low ROB in all ROBIS domains (Table 2).

Table 2.

Mean section adherence for each domain of ROBIS tool and difference between studies at high risk and low risk.

ROBD1D2D3D4Overall
High  Low  Differences  High  Low  Differences  High  Low  Differences  High  Low  Differences  High  Low  Differences 
PRISMA  Adherence in percentage (%)
Title  98.9  100.0  −1.1  98.9  100.0  −1.1  99.0  100.0  −1.0  98.9  100.0  −1.1  99.3  100.0  −0.7 
Abstract  59.8  54.7  5.1  48.4  74.7  −26.3  64.6  56.9  7.7  58.9  56.3  2.6  61.5  67.9  −6.4 
Introduction  70.7  78.9  −8.3  71.0  79.7  −8.8  75.3  74.3  0.9  73.3  76.8  −3.4  73.7  76.8  −3.1 
Methods  42.4  61.1  −18.7  41.6  63.7  −22.1  41.3  61.4  −20.1  42.5  61.8  −19.3  43.9  66.1  −22.2 
Results  57.0  71.4  −14.4  57.0  70.0  −13.0  56.5  71.1  −14.6  54.2  72.4  −18.2  58.2  72.2  −14.0 
Discussion  76.4  92.7  −16.3  77.8  88.6  −10.8  75.1  92.1  −17.0  75.9  91.5  −15.6  79.3  93.5  −14.2 
Funding  38.0  45.3  −7.3  40.9  39.2  1.6  43.4  40.3  3.2  40.0  38.0  2.0  39.3  42.9  −3.6 
Overall Adherence  55.0  69.6  −14.6  54.6  70.3  −15.7  55.0  69.0  −14.0  54.5  69.6  −15.1  56.6  72.1  −15.5 

Abbreviations: D1, concerns regarding specification of study; D2, concerns regarding methods used to identify and/or select studies eligibility criteria; D3, concerns regarding methods used to collect data and appraise studies; D4, concerns regarding the synthesis and findings; Overall, overall Risk of Bias

The univariable linear regression analysis showed a negative association between high ROB and completeness of the reporting (adherence to the PRISMA checklist) in all domains (see supplementary material Table S.6). Still, when we entered all independent variables in the same model, the multivariable linear regression analyses (Table 3 – primary analysis) reported an association between completeness of reporting and ROB in domains 1, 2, and overall ROB.

Table 3.

Multivariable linear regression models for primary and secondary study analysis with association between independent variables and overall adherence to the PRISMA Checklist.

  B (95%CI)  Std. Error B  β  P value 
  Primary analysis
Constant  74.7 (71.2, 78.1)  1.7    <0.001* 
Risk of bias in D1 (Low vs High)  −5.7 (−10.1, −1.3)  2.2  −0.2  0.011* 
Risk of bias in D2 (Low vs High)  −5.4 (−9.7, −1.1)  2.2  −0.2  0.015* 
Risk of bias in D3 (Low vs High)  −4.0 (−8.7, 0.8)  2.4  −0.1  0.105 
Risk of bias in D4 (Low vs High)  −3.0 (−7.8, 1.7)  2.4  −0.1  0.211 
Overall risk of bias (Low vs High)  −7.1 (−12.1, −2.0)  2.6  −0.2  0.006* 
R2=0.314
  Secondary analysis
Constant  56.9 (49.0, 65.7)  4.0    <0.001* 
Quartile (Q1 vs. Q2)  −1.6 (−7.6, 4.5)  3.1  −0.04  0.610 
Quartile (Q1 vs. Q3)  −0.4 (−6.4, 5.5)  3.0  −0.01  0.884 
Quartile (Q1 vs. Q4)  −7.2 (−13.2, −1.3)  3.0  −0.2  0.017* 
Year of publication  0.4 (−0.3, 1.2)  0.4  0.1  0.263 
Publication Options (Hybrid vs. Open Access)  3.4 (−2.7, 9.5)  3.1  0.1  0.269 
Protocol Registration (No vs. Yes)  11.9 (5.9, 18.0)  3.0  0.3  <0.001* 
R2= 0.135

Note: values with * and in bold are statistically significant. Dependent variable: mean overall adherence to the CONSORT checklist

Abbreviations: B, unstandardized Beta coefficient; β, standardized Beta coefficient; D1, concerns regarding specification of study; D2, concerns regarding methods used to identify and/or select studies eligibility criteria; D3, concerns regarding methods used to collect data and appraise studies; D4, concerns regarding the synthesis and findings; Low, low risk of bias; High. high risk of bias; Overall ROB, overall risk of bias; Q, quartile.

Having a high ROB in domain 1 (B= -5.7%; 95%CI -10.1, -1.3) and domain 2 (B= -5.4%; 95%CI -9.7, -1.1) are significant predictors of lower overall adherence to the PRISMA checklist. If we look at the overall ROB, having a high overall ROB is a significant predictor of lower adherence (B=-7.1%; 95%CI -12.1, -2.0).

Relationship between completeness of reporting and studies and journal characteristics

Univariable linear regression models showed a significant relationship between journal IF, publication year, study protocol registration, and completeness of the reporting (see supplementary material Table S.7). When we entered all independent variables in the multivariable linear regression model, only journal IF and protocol registration were significant predictors (see Table 3 – secondary analysis). Studies published in fourth quartile journals displayed a lower overall adherence (B= -7.2%; 95%CI -13.2, -1.3) than those published in first quartile journals; the overall adherence increased (B= 11.9%; 95%CI 5.9, 18.0) if the SR protocol was registered.

Discussion

This study evaluated the completeness of reporting in a sample of SRs published in rehabilitation journals, and the existing correlations with ROB and other important characteristics were investigated. Our results showed that the completeness of the reporting was suboptimal: in fact, the overall adherence to the PRISMA checklist was 61.4%, with a range from 100% to 16% (see supplementary material Figure S.1). These results are in line with others in different biomedical research fields such as vascular surgery,12 emergency medicine,5 and nursing21 that showed a 59% to 70% range of adherence to the PRISMA 2009.

To our knowledge, this is the first study to compare the completeness of reporting to the ROB in the rehabilitation field. Completeness of the reporting showed a relationship with the ROB: the higher the ROB, the poorer the completeness of reporting. However, the potential causality in this association remains to be proven. We investigated this relationship through multivariable regression analysis (Table 3) with all ROB domains and overall ROB score in the same model. We adopted this solution to clearly show how reporting can be influenced by both individual domains (especially domains 1 and 2) and the overall score without multicollinearity. Tunis et al.22 evaluated the same association in SRs published in major radiology journals. Despite differences in methods concerning our study (smaller sample size and the use of the AMSTAR tool23), they found a strong association between completeness of reporting and higher study quality.

Our secondary analysis showed that completeness of reporting is also associated with the IF and registration of the study protocol. Although IF metrics do not indicate the quality of the studies published in a journal,24 our results suggest that reporting is more complete in journals with a high IF. Completeness of reporting is strongly associated with the registration of the study protocol: there was an 11.95% increase in overall adherence to the PRISMA checklist if the protocol was registered.

Publication of the study protocol is one of the most critical aspects that emerge from our study. Only 15.5% (31/200 – see supplementary material Table S.3) of the studies included had published the protocol. Item 5 of the PRISMA statement was the least reported item. This remarkable observation is also confirmed by other studies. In gastroenterology and hepatology journals it was reported in only 4.4% of studies,25 while it was 27% in orthodontic journals,26; 15% of the SRs in radiology journals22 mentioned research protocols. This happens despite methodological standards explicitly requiring the study protocol registration and publication in SRs.27

There is much room for improvement for all actors in the evidence production process (i.e. editors, peer-reviewers and researchers). Journal editors should personally check the peer-reviews and encourage peer-reviewers to pay more attention to the reporting issues. Peer-reviewers should check the accuracy of the reporting in the studies, while researchers need to improve compliance with the reporting standard and increase the readability and transparency of their research reports. Moreover, the researchers play an important role in this topic. They should not simply report their manuscript following the reporting guidelines (e.g. the PRISMA statement for systematic review authors) but they should provide appendices and raw data in sufficient detail to allow peer-reviewers and readers to assess their manuscripts and to make their results applicable.28

Despite some strategies adopted by journals to improve RGs use (such as requiring authors to follow the RG in the ‘‘Instructions to authors’’), these do not appear to be consistently effective.29,30 Journal editors may need to adopt electronic systems (e.g. through artificial intelligence31) to check reporting accuracy in a scientific paper. These systems may run checks during the submission process, detecting and prompting for corrections before submission and explicitly requiring that authors follow RGs indications.

Study limitations

We used the PRISMA checklist to assess the completeness of reporting, despite the checklist being a guide for writing and not for evaluating quality. We are aware that despite being consistent with previous meta-research studies,12,13,22 this may represent a limitation.32 To overcome this issue, we have tried to define how we have reached the scores clearly, and we ran a preliminary pilot test to increase inter-assessor agreement, especially on items requiring interpretation (sometimes indicated by phrases like “if relevant” or “if applicable”). Moreover, between 2011 and 2020 many other SRs have been published and our sample (200 hits) may not be representative of all studies published in such time frame. However, our choice is based on the sample size used in other similar meta-research studies12-14,33 and we believe it is representative of the characteristic of the SRs published in the rehabilitation journals.

Conclusions

The completeness of reporting in SRs published in rehabilitation journals is suboptimal. High ROB is associated with poorer completeness of reporting, while the registration of study protocols and journal ranking are also related to more complete reporting. Journal editors, peer-reviewers, and researchers play an essential role in increasing the readability and transparency of the research reports; as such, they should make every effort to improve the quality of research by adopting strategies (e.g. electronic systems and artificial intelligence) to check reporting accuracy in scientific papers.

Acknowledgements

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References
[1]
I Chalmers, K Dickersin, TC. Chalmers.
Getting to grips with Archie Cochrane's agenda.
Br Med J, 305 (1992), pp. 786-788
[2]
M Paul, L. Leibovici.
Systematic review or meta-analysis? Their place in the evidence hierarchy.
Clin Microbiol Infect, 20 (2014), pp. 97-100
[3]
D Moher, J Tetzlaff, AC Tricco, M Sampson, DG. Altman.
Epidemiology and reporting characteristics of systematic reviews.
[4]
D Moher, A Liberati, J Tetzlaff, DG. Altman.
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
Ann Intern Med, 151 (2009), pp. 264-269
[5]
F Nawijn, WHW Ham, RM Houwert, RHH Groenwold, F Hietbrink, DPJ. Smeeing.
Quality of reporting of systematic reviews and meta-analyses in emergency medicine based on the PRISMA statement.
BMC Emergency Med, 19 (2019), pp. 19
[6]
T Horsley, J Galipeau, J Petkovic, J Zeiter, SJ Hamstra, DA. Cook.
Reporting quality and risk of bias in randomised trials in health professions education.
Med Educ, 51 (2017), pp. 61-71
[7]
DG Altman.
Importance of transparent reporting of health research.
Guidelines for Reporting Health Research: A User's Manual, pp. 3-13
[8]
I Chalmers, P. Glasziou.
Avoidable waste in the production and reporting of research evidence.
Obstet Gynecol, 114 (2009), pp. 1341-1345
[9]
D Moher, DJ Cook, S Eastwood, I Olkin, D Rennie, DF. Stroup.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses.
Lancet, 354 (1999), pp. 1896-1900
[10]
MJ Page, JE McKenzie, PM Bossuyt, et al.
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.
BMJ, 372 (2021), pp. n71
[11]
P Whiting, J Savović, JPT Higgins, et al.
ROBIS: A new tool to assess risk of bias in systematic reviews was developed.
J Clin Epidemiol, 69 (2016), pp. 225-234
[12]
WK Tan, J Wigley, S. Shantikumar.
The reporting quality of systematic reviews and meta-analyses in vascular surgery needs improvement: a systematic review.
Int J Surg, 12 (2014), pp. 1262-1265
[13]
JJ Gagnier, PJ. Kellam.
Reporting and methodological quality of systematic reviews in the orthopaedic literature. Review.
J Bone Joint Surgery - Series A, 95 (2013), pp. e771-e777
[14]
T Innocenti, S Salvioli, S Giagio, D Feller, N Cartabellotta, A. Chiarotto.
Declaration of use and appropriate use of reporting guidelines in high-impact rehabilitation journals is limited: a meta-research study.
J Clin Epidemiol, 131 (2021), pp. 43-50
[15]
Journal Citation Report.
[16]
Innocenti T, Feller D, Giagio S, et al. Adherence to the PRISMA statement and risk of bias assessment in Systematic Reviews in rehabilitation journals: a protocol for a meta-research study. medRxiv. 2021:2021.01.21.21250260. https://doi.org/10.1101/2021.01.21.21250260
[17]
DO Lawson, L Puljak, D Pieper, et al.
Reporting of methodological studies in health research: a protocol for the development of the MethodologIcal STudy reportIng Checklist (MISTIC).
[18]
JP. Ioannidis.
The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.
Milbank Q, 94 (2016), pp. 485-514
[19]
M Krnic Martinic, D Pieper, A Glatt, L Puljak.
Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks.
BMC Med Res Methodol, 19 (2019), pp. 203
[20]
LA McGuinness, JPT. Higgins.
Risk-of-bias VISualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments.
Res Synth Methods, (2020),
[21]
WWS Tam, KKH Lo, P. Khalechelvam.
Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study.
[22]
AS Tunis, MD McInnes, R Hanna, K. Esmail.
Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement?.
Radiology, 269 (2013), pp. 413-426
[23]
BJ Shea, JM Grimshaw, GA Wells, et al.
Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
BMC Med Res Methodol, 7 (2007), pp. 10
[24]
M Saginur, D Fergusson, T Zhang, et al.
Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis.
[25]
N Panic, E Leoncini, G de Belvis, W Ricciardi, S. Boccia.
Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses.
[26]
PS Fleming, J Seehra, A Polychronopoulou, Z Fedorowicz, N. Pandis.
A PRISMA assessment of the reporting quality of systematic reviews in orthodontics.
Angle Orthod, 83 (2013), pp. 158-163
[27]
Cochrane Handbook for Systematic Reviews of Interventions Version 6.2 (updated February 2021),
[28]
S Giagio, T Innocenti, S Salvioli, et al.
Completeness of exercise reporting among randomized controlled trials on pelvic floor muscle training for women with pelvic organ prolapse: a systematic review.
Neurourol Urodyn, 40 (2021), pp. 1424-1432
[29]
D Blanco, AM Biggane, E Cobo, et al.
Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers?.
[30]
D Blanco, D Altman, D Moher, I Boutron, JJ Kirkham, E. Cobo.
Scoping review on interventions to improve adherence to reporting guidelines in health research.
[31]
A Checco, L Bracciale, P Loreti, S Pinfield, G. Bianchi.
AI-assisted peer review.
Humanit Soc Sci Commun, 8 (2021), pp. 25
[32]
P Logullo, A MacCarthy, S Kirtley, GS. Collins.
Reporting guideline checklists are not quality evaluation forms: they are guidance for writing.
Health Sci Rep, 3 (2020), pp. e165
[33]
L Caulley, F Catala-Lopez, J Whelan, et al.
Reporting guidelines of health research studies are frequently used inappropriately.
Copyright © 2022. The Author(s)
Idiomas
Brazilian Journal of Physical Therapy

Subscribe to our newsletter

Article options
Tools
Supplemental materials
en pt
Cookies policy Política de cookies
To improve our services and products, we use "cookies" (own or third parties authorized) to show advertising related to client preferences through the analyses of navigation customer behavior. Continuing navigation will be considered as acceptance of this use. You can change the settings or obtain more information by clicking here. Utilizamos cookies próprios e de terceiros para melhorar nossos serviços e mostrar publicidade relacionada às suas preferências, analisando seus hábitos de navegação. Se continuar a navegar, consideramos que aceita o seu uso. Você pode alterar a configuração ou obter mais informações aqui.