This is a measurement property study of the modified Fresno test – Brazilian-Portuguese short version.22 We followed the recommendations from the COnsensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) initiative.23-25

A convenience sample of physical therapists was invited to participate in the study. The sample size was estimated as proposed by the COSMIN initiative for measurement properties tests.23-25 Inclusion criteria were: (1) physical therapist professors from public and private institutions involved in clinical practice, and (2) physical therapist specialists in any clinical practice areas. Participants were excluded from the study if they refused to answer the test and if they started to answer but did not finish the online survey. The Universidade Cidade de São Paulo local research ethics committee (number #3.636.011/2019) approved the study. This measurement property study was conducted through an online survey using the SurveyMonkey platform. The study was advertised through direct e-mail and text messages to potential participants and posts on social networks (Instagram, Facebook, and Linked In).

The following sociodemographic variables were collected for each participant: age, sex, institution where they graduated (private or public funded), time (years) since graduation, and qualification. The SurveyMonkey platform recorded the time spent by participants to complete the test.

Data were collected using the modified Fresno Test - Brazilian-Portuguese short version with 9-items, proposed by Silva et al.22 The test features an entry-level text with completion instructions and two clinical scenarios. The participant must choose one of the clinical scenarios proposed by the test and have a maximum of 1 hour to answer all 9 items.2,17 The total score of the modified Fresno test – Brazilian-Portuguese short version is calculated by summing up the answers from each item. For example, item 2 asks the participant to name the research databases to answer the chosen clinical scenario, which has the following scoring criteria: (a) variety of sources, (b) convenience, (c) clinical relevance, and (d) validity.2,17 The answer to each item is scored on a 5-point Likert scale: (1) not evident, (2) limited, (3) minimal, (4) strong, and (5) excellent. The sum of the scores for each criterion results in a score per question that varies between 0 and 24 points: questions Q1 to Q6 allow a maximum score of 24 points for each question; for question Q7, the maximum score is 16 points, and for questions Q8 to Q9, a maximum score of 4 points per question. The total score is the sum of points for all questions and ranges from 0 to 168 points, with a higher score meaning a better understanding of EBP.22

Each participant answered the questionnaire twice (i.e., test-retest design), at the initial assessment (first assessment) and 7 days later (second assessment). At the first assessment, participants received instructions on how to answer the questionnaire and they completed the participant informed consent form. After agreeing to participate in the study and choosing one of the clinical scenarios presented, the participants had up to 1 hour to complete the questionnaire. After this first assessment, all participants were asked to answer the test again seven days later according to the instructions described in the questionnaire (second assessment). The instrument's total score was calculated by two independent assessors.22

Two assessors with experience in EBP analyzed the answers and calculated the total score of the questionnaire. They received a 3-hour training session about the scoring of the modified Fresno test – Brazilian-Portuguese short version. The first hour of the training was dedicated to explaining the questionnaire's scoring criteria. In the second hour of the training, a pilot test was conducted with each rater scoring one questionnaire. Finally, the third hour was dedicated to analysing and discussing the results of the pilot test. After the training, the assessors randomly scored each questionnaire blinded to the participants’ details. Each assessor received the questionnaires after the first and second assessments. Responsiveness of the instrument was analyzed including participants who answered the questionnaire again four months after the first assessment. Our hypothesis is that changes in effect size will be detected after 4 months following the administration of the questionnaire. The effect is related to the fact that when using the modified Fresno test – Portuguese-Brazilian short version, the participant will benefit from the training and access to the information contained therein, providing learning and the search for new knowledge for the use of EBP.22

Descriptive statistics, including measures of central tendency (simple and relative frequencies, mean and median) and measures of dispersion (standard deviation), were used to describe the sociodemographic data, as well as the total scores and the score for each of the dimension (i.e., confirmatory factor analysis) of the modified Fresno test – Brazilian-Portuguese short version.22

For reliability and agreement we compared the score for each item and the total score of the test obtained in the first and second assessments for each assessor (intra-rater reliability) and between assessors for the score obtained in the first assessment (inter-rater reliability) with a 7-day interval. The reliability analyses followed the same method used in the analysis of the original instrument using intraclass correlation coefficient (ICC) of type 2,1 for continuous variables.2 The ICC was classified according to Fleiss (1986)26: poor (<0.4), good (0.4–0.75), and excellent reliability (>0.75). The Standard Error of Measurement (SEM) was used to assess agreement.13 The percentage of the SEM related to the total score of a questionnaire was interpreted according to Ostelo et al.27 as ≤ 5 %, very good; > 5 % and ≤ 10 %, good; > 10 % and ≤ 20 %, doubtful; and > 20 %, negative.27 Internal consistency analysis was performed by calculating the Cronbach's alpha coefficient for all items of the questionnaire and the corrected item-total correlations on data from assessor 1 collected at the first and second assessment.28 Alpha value of ≥ 0.7 was defined as “acceptable” and ≥ 0.80 as “good.” A value greater than 0.2 was considered for the corrected item-total correlations.29

Content validity was analyzed through the evaluation of a committee of 16 experts (physical therapists, statisticians, and epidemiologists), considering the content validity index. The content validity index measures the percentage of agreement among the members of the expert committee on the aspects addressed in the test items on items of the modified Fresno Test – Brazilian-Portuguese short version.22 The acceptable agreement index among the expert committee members was 0.80. Confirmatory factor analysis was performed using the principal components method for structural validity.30 The total score of the 266 responses in the test and retest was evaluated, totaling 1064 observations. The confirmatory factor analysis was used to investigate the adequacy of the modified Fresno test – a Brazilian-Portuguese short version, with 9 items, represented as “model 2”, with the model proposed by Silva et al.22 of 13 items, defined as “model 1”. The first model evaluated the questionnaire's structure composed of 3 (three) factors. The second model tested the hypothesis that the questionnaire is composed of 2 (two) factors. The maximum likelihood method was used for the analysis. The following indices were used for model fitting: chi-square index, where values with significance levels p < 0.05 are estimated. The Goodness of Fit Index indicates how much the proposed models explain variance and covariance. The comparative fit index, the normed fit index, and the non-normed fit index, representing the proportional improvement in model fit, were also analyzed. Values greater than 0.90 for these indices indicate an adequate model. The root mean square error of approximation considers the error of approximation in the population in a covariance matrix. Values of 0.08 or less represent a reasonable error. The expected cross-validation index, which indicates the best fit of the models and is appropriate for comparing ungrouped models, was also analyzed. There are no benchmark values to rank model fit, and it should be as low as possible.

The responsiveness of the modified Fresno test - Brazilian-Portuguese short version was analyzed by calculating the effect size.22 The effect size is the difference between the mean total score at the 4-month minus the mean total score at baseline (i.e., first assessment) divided by the standard deviation of the baseline scores. The effect size was interpreted as 0.01 as very small, 0.2 as small, 0.5 as medium, 0.8 as large, 1.2 as very large, and 2.0 as huge.31

To analyze the ceiling and floor effect, the assessor's percentage of minimum and maximum possible scores in the first and second assessments were calculated. Ceiling and floor effects were considered present when >15 % of respondents achieved the lowest (i.e., 0 points) or highest (i.e., 168 points) possible total score. The presence of ceiling effects may indicate that the items are not challenging enough or too challenging, respectively.23 The Statistical Package for the Social Sciences (SPSS IBM® - version 22.0) software was used for the statistical treatment of the data.

Eligible physical therapists were recruited for this study between January 2020 and July 2021. A total of 133 physical therapists responded to the questionnaire at the first and second assessments and were included in the reliability analysis. Of these, 82 (62 %) participants answered the questionnaire again four months after the initial assessment and were included in the responsiveness analysis. Participants who were missing at four months did not complete the questionnaire due to lack of knowledge or disinterest in the study.

Table 1 presents the sociodemographic characteristics of the study participants. The sample consisted mainly of female participants (66.9 %), predominantly between 30 and 39 years old (46.6 %). Most of them have a maximum qualification of a master's degree (36.1 %), having graduated from a private educational institution (79.7 %) and completed the degree between 6 and 10 years ago (27.1 %).

The intra-rater reliability for assessor 1 showed ICCs for each item ranging from 0.63 to 1.0, and SEM ranged from 0.11 to 2.76. Intra-rater reliability was rated as excellent for 8 out of the 9 items. The only item that showed good intra-rater reliability was item 2. The ICCs for most of the items were higher for assessor 2 compared to the ones found for assessor 1. For assessor 2, ICCs for each item ranged from 0.88 to 0.99, and SEM ranged from 0.34 to 2.12 points. These results should be interpreted as excellent intra-rater reliability for each item rated by assessor 2.

For the total score of the modified Fresno test – short version, the intra-rater reliability was rated as excellent for both assessors. Assessor 1 and assessor 2 showed ICCs values of 0.95 (95 % CI: 0.94, 0.97) and 0.98 (95 % CI: 0.98, 0.99), respectively. Inter-rater reliability was rated as excellent, with ICC of 0.93 (95 % CI: 0.91, 0.95). The SEMs were 10.38 points for the inter-rater reliability, 8.72 points for the intra-rater reliability of assessor 1, and 4.52 points for the intra-rater reliability of assessor 2. The SEMs were classified as very good (values ≤ 5 % of the total score) (Table 2).

The internal consistency obtained with Cronbach's alpha indicated a value of 0.93 for the nine items of the instrument. The α coefficients calculated for the corrected item-total correlations showed values greater than 0.40.

Content validity was assessed using the content validity index, which measures the proportion or percentage of agreement among the expert committee on the questionnaire items. A content validity index of 0.96 was found.

The confirmatory factor analysis showed the “Model 2”; with nine items and two factors provided a better fit compared to "Model 1″ with 13 items and three factors (Fig. 1).

Fig. 1.

Confirmatory factor analysis diagram (CFA).

(0.16MB).

The goodness of fit index and comparative fit index reached an acceptable level (≥ 0.90). “Model 2”, which tested a two-factor structure, provided a better fit to the data (Table 3). However, the fit indices of the second model also did not reach acceptable levels. Therefore, the second model, although better than the first model, did not adequately fit the data.

The responsiveness analysis of the modified Fresno test - Brazilian-Portuguese short version is presented in Table 4, with the results of the effect sizes of the mean differences of the total scores in the first and second stages. The table analyzed the effect size appropriate for the paired t-test. For the total score, the effect size was 0.13 points and rated as “very small,” i.e., 4-month follow-up minus the baseline.

No ceiling and floor effects were found among respondents.