This is a study of measurement properties that employed both cross-sectional and prospective data. The study received approval from the Human Research Ethics Committee of the Federal University of Santa Maria (Santa Maria, RS, Brazil; registration no CAAE 48078321.0.0000.5346), and informed consent was obtained from all participants. The authors of the original versions granted prior authorization for the translation process. The study was conducted and reported in accordance with established recommendations and guidelines.13–15,32,33

The questionnaires were translated following the guidelines recommended by Beaton et al.32 and the COSMIN checklist.15 The translation process comprised the following steps:

Translation: Two independent bilingual translators translated the S-EBPQ from English into Brazilian Portuguese, and another two independent bilingual translators translated the EBP-COQ from Spanish into Brazilian Portuguese. The translators were native speakers of Brazilian Portuguese, and for each pair, one translator was literate in EBP, while the other had no background in this health field.

Synthesis: The translated versions were compared, and the translators reached a consensus to produce a single Brazilian Portuguese version of each questionnaire.

Back Translation: Two additional pairs of independent bilingual translators, unaware of the original versions and having no background in EBP, back-translated the Brazilian Portuguese versions of the S-EBPQ and EBP-COQ into English and Spanish, respectively. One of the back-translators of the EBP-COQ was a native Spanish speaker.

Expert Committee Review: An expert committee of experienced researchers in questionnaire translation convened by the translators, the authors of the original versions, and healthcare students (physical therapy n=2, nursing n=2, occupational therapy n=1), reviewed all versions to address discrepancies and culturally adapt the questionnaires. This process resulted in the prefinal Brazilian Portuguese versions of the S-EBPQ and EBP-COQ, named “S-EBPQ-BR” and “EBP-COQ-BR for nursing students”.

Subsequently, the committee modified the nursing-specific questions within the EBP-COQ-BR for nursing students to enhance applicability to a broader range of healthcare students, resulting in the version "EBP-COQ-BR for healthcare students". The S-EBPQ-BR remained unchanged in this regard.

Pretesting: The three prefinal versions (S-EBPQ-BR, EBP-COQ-BR for nursing students, and EBP-COQ-BR for healthcare students) underwent pretesting to assess comprehensibility.

Student Evidence-Based Practice Questionnaire in Brazilian Portuguese (S-EBPQ-BR): the tool consists of 21 items across four subscales: practice, attitudes, evidence retrieval and review, and EBP application.16 Each item is rated on a scale of one to seven. Total or subscale scores are calculated by averaging,16 which results in scores range from one to seven, with higher scores reflecting greater EBP competence.

Evidence-Based Practice Evaluation Competence Questionnaire in Brazilian Portuguese (EBP-COQ-BR): the tool consists of 25 items across three subscales: attitudes, skills, and knowledge.17 Each item is rated on a scale of one to five. Total or subscale scores are calculated by averaging,17 which results in scores range from one to five, with higher scores reflecting greater EBP competence. We decided to include both versions for nursing and healthcare students to evaluate the validity of the original version (nursing) and to investigate whether the adaptations for healthcare students affected the measurement properties.

Evidence-Based Practice Questionnaire (EBPQ): this questionnaire was validated to evaluate EBP competencies among nurses,34 and was included to evaluate construct validity. The tool consists of 24 items across three subscales: practice, attitudes, and knowledge.29,34 Each item is rated on a scale of one to seven. Total or subscale scores are calculated by averaging,29,34 which results in scores range from one to seven, with higher scores reflecting greater EBP competence. This instrument has already been translated and validated in Brazilian Portuguese, presenting adequate measurement properties.29

Participants completed the questionnaires and provided demographic data, including age, course, and course stage. Nursing students completed the EBPQ, S-EBPQ-BR, and EBP-COQ-BR for nursing students, while other healthcare students completed the EBPQ, S-EBPQ-BR, and EBP-COQ-BR for healthcare students. To assess questionnaire reliability, participants completed the S-EBPQ-BR and EBP-COQ-BR again one week later.

A subgroup of physical therapy students completed the S-EBPQ-BR and EBP-COQ-BR for healthcare students one month after the first assessment to evaluate questionnaire responsiveness (Fig. 1). These participants underwent a month-long training program, which consisted of 30 h of both online and in-person activities designed to enhance their EBP competencies.10,11 The content of the program was derived from the book authored by Herbert et al.1 and encompassed instructions on the definition and fundamental concepts of EBP, techniques for searching scientific literature, criteria for evaluating robust evidence, foundational principles of statistical analysis and data presentation, and interpretation of the clinical significance of scientific findings.

Fig. 1.

Flow diagram. (Abbreviations: EBP-COQ-BR, evidence-based practice evaluation competence questionnaire in Brazilian portuguese; EBPQ, evidence-based practice questionnaire in Brazilian portuguese; S-EBPQ-BR, student evidence-based practice questionnaire in Brazilian portuguese.)

(0.33MB).

All participants completed the questionnaires using either paper format or an online survey tool (QuestionPro, QuestionPro Inc.). Previous studies have demonstrated the comparability of both methods,35,36 and we verified this comparability in this study through testing (Supplementary material 1A). Data collection was conducted between June 2021 and June 2023.

We included undergraduate students enrolled in healthcare courses from public or private universities, including nursing, physical therapy, occupational therapy, speech therapy, biomedical science, dentistry, gerontology, medicine, and nutrition. Eligible participants needed to be fluent in Brazilian Portuguese and over 18 years old. No restriction was imposed regarding their curricular stage. We recruited a convenience sample by directly inviting students on university campuses and through social media posts that provided a link to the online survey.

To determine the sample size, we adhered to guidelines recommending the inclusion of at least 100 participants to ensure sufficient statistical power for the analyses.32 However, for specific analyses, we considered smaller sample sizes. In the responsiveness analysis, we predetermined the sample size based on a pre-post effect size of 0.5, which we deemed to be the smallest worthwhile difference,37,38 with power of 80% and an alpha of 0.05. The calculation indicated that a minimum of 34 participants was required. For the reliability analysis, we predetermined the sample size based on a power of 80%, an alpha of 0.05, an intraclass correlation coefficient (ICC3,1) of 0.5 as the minimum acceptable value (ρ0) and an ICC3,1 estimated of 0.7 (ρ1); the calculation indicated that a minimum of 79 participants was required.39

Pre-Testing Phase: in the pretest versions, an open field was provided after each item to allow participants to comment on the applicability of the question or any difficulties in comprehension. Additionally, participants were asked to indicate whether the question was “clear” or “unclear” and rate their understanding of each item on a scale from zero (totally unclear) to 10 (completely understood). The expert committee analyzed the responses to identify any concerns raised by participants that could indicate content-related issues.14,15

Internal Consistency: we employed Cronbach's alpha to evaluate the degree of interrelatedness among the questionnaire items.14,15 Values between 0.70-0.95 are indicative of strong internal consistency.13,40

Reliability: we used the intraclass correlation coefficient (ICC3,1) to assess the instruments’ reproducibility and their relationship with measurement error.14,15 Reliability was categorized based on the ICC value as follow: <0.50= poor reliability, 0.50-0.75=moderate reliability, >0.75-0.90=good reliability, and >0.90=excellent reliability.41

Measurement Error: the standard error of measurement (SEM) was employed to quantify the extent of error based on the observed variability.13–15 To calculate the SEM, we used the formula SD√1-ICC2,1, where SD represents the standard deviation, encompassing both test and retest.42 The percentage of SEM in relation to the total score was used to classify the agreement: ≤5%=very good, 5-10%=good, 11-20%=doubtful; and >20%=negative.43 Additionally, we conducted an analysis of the minimum detectable change (MDC) using the formula MDC95=1.96 x SEM x √2.13 Formulas are presented in Supplementary material 2.

Construct Validity: Pearson correlation (95% CI) was utilized to assess the relationship between the construct of the tested instruments and the EBPQ. We also examined overlapping subscales between the tested instruments and the reference (EBPQ). Correlation strength was categorized as very weak (r=0.0-0.19), weak (r=0.20-0.39), moderate (r=0.40-0.59), strong (r=0.60-0.79), and very strong (r≥0.80).44,45 The construct validity analysis was based on hypothesis testing; we hypothesized that total scores and the subscales of the tested instruments would exhibit convergent (positive) correlations of at least moderate strength (r≥0.40) with the reference instrument.46,47

Responsiveness: to assess whether the tested instruments could detect longitudinal changes,13–15,40,48 we employed the paired t-test, along with mean difference (95% CI) and effect size (Cohen's d).49 Given the established effectiveness of the education program,10,11 we hypothesized that the questionnaires would demonstrate an effect size of at least moderate magnitude (d=0.5)48 associated with significant differences.

Ceiling and floor effect: to evaluate the sensitivity of the instruments to detect various levels of EBP competencies, we examined the percentage of participants reporting the minimum or maximum scores.13–15 Percentages lower than 15% indicated the absence of ceiling and floor effects.13,40

We used R statistical software version 4.1.1 (packages psych, stats and mice) for analyses. Questionnaires with less than half of their items answered were excluded. Missing data were addressed using multiple imputation, generating 10 datasets with up to 50 iterations based on the responses (Fig. 1). Results were stratified according to the cohort (nursing and other healthcare students). For the S-EBPQ-BR, all participants were analyzed as one cohort because the questionnaire was identical for all cohorts. Given that the tested tools exhibited multidimensionality in their construct, the analyses utilized both the total scores of the questionnaires and their respective subscales.

A total of 491 students were recruited. Of these, 31 participated in the pretest phase, and 442 were included in the main analysis. Eighteen students were excluded due to incomplete responses (Fig. 1). The individual participant data are presented in Supplementary material 2 and the sample characteristics for each analysis are presented in Table 1.

For the S-EBPQ-BR, no adaptations were necessary. For the EBP-COQ-BR for nursing students, we incorporated sex-specific words in item A6. In the case of the EBP-COQ-BR for healthcare students, we made the following modifications: we replaced terms associated with “nursing” with “healthcare professional” in items A3, A4, and A6; we replaced the term “nursing” with “my field of specialty in health” in item A13; and we replaced the databases Joanna Briggs and Evidence Based-Nursing, with PubMed and Medline in item C2 (Supplementary material 1B).

During the pretest analysis, 31 healthcare students, including nursing students, completed the questionnaires. The percentage of participants classifying items as “clear” varied between 90 and 97% for the S-EBPQ-BR, 87% and 97% for the EBP-COQ-BR for nursing, and 87% and 97% for the EBP-COQ-BR for healthcare students. For the rating scale ranging from 0 (totally unclear) to 10 (completely understood), the mean scores varied between 8 and 10 for all pretest items. The comments included in the open fields did not indicate any meaningful issues related to understanding or applicability (Supplementary material 1B). The final versions of the questionnaires are presented in Appendix.

The instruments tested demonstrated strong internal consistency, as evidenced by Cronbach's alpha values, which ranged from 0.87 to 0.92 (Table 1). The exception was the “attitude” subscale for the S-EBPQ-BR (Table 2).

Most of the instruments exhibited good or moderate reliability (ICC3,1≥0.54), with the exception being the “attitude” subscale of the S-EBPQ-BR for healthcare students, which displayed poor reliability (ICC3,1=0.44; Table 1). The SEM ranged from 0.2 to 0.7 points, and all instruments demonstrated good to very good agreement (4-10%) (Table 2). The MDC95 for the EBP-COQ-BR versions ranged from 0.5 to 1.2 points (10-24%), while those for the S-EBPQ-BR versions ranged from 0.9 to 2.1 points (13-30%; Table 2).

Most of the analyses supported our hypotheses, revealing moderate to very strong correlations with the EBPQ scores (Table 3). The “attitude” subscale of both versions of the EBP-COQ-BR and “skills” subscale of the EBP-COQ-BR for nursing exhibited weak correlations with the corresponding subscales of the EBPQ (Table 3).

Table 3.

Construct validity analyses for the Brazilian Portuguese versions of the tested questionnaires.

Abbreviations: CI, confidence interval; EBP-COQ-BR, Evidence-Based Practice Evaluation Competence Questionnaire in Brazilian Portuguese; EBPQ, Evidence-Based Practice Questionnaire in Brazilian Portuguese; S-EBPQ-BR, Student Evidence-Based Practice Questionnaire in Brazilian Portuguese.

*No analysis was performed for the grey cells.

A ceiling effect was observed for the “attitude” subscale of the S-EBPQ-BR in all analysis, with no instrument displaying a floor effect (Table 2).

Both the EBP-COQ-BR for healthcare students and S-EBPQ-BR questionnaires exhibited significant differences in their scores before and after an educational program, with effect sizes larger than 0.5. These differences were associated with moderate to very large effect sizes (Table 4).

The validation process for the versions targeting healthcare students involved participants from various courses. While this approach allows for the generalization of the findings, it may obscure specific characteristics of different healthcare fields. Assessors should be mindful of this potential limitation if they intend to assess a particular student population within healthcare. Additionally, the assessment of responsiveness was conducted solely among physical therapy students participating in a specific EBP course. Although the responsiveness is clear, it is important to acknowledge that students from other healthcare courses may respond differently, potentially affecting the questionnaires' ability to detect changes. Another limitation concerns the absence of structural validity analysis. Despite our sample size ensuring statistical power for the analyses conducted, it does not provide sufficient confidence to perform confirmatory factor analysis for all analyzed questionnaires.53 To avoid potentially distorted results due to small sample sizes, we chose not to include it in the study. Although the structure of the questionnaires has been investigated in previous studies,18,19,21,25–27 further investigation can determine whether the Brazilian Portuguese versions maintain their original structural integrity. Lastly, the back-translation step was performed mostly by non-native speakers of the instruments’ original languages, which should be acknowledge as a limitation.32