The Pediatric Rehabilitation Intervention Measure of Engagement - Observation Version (PRIME-O) is a standardized tool that assesses client engagement, which can be the child, youth, or parent/caregiver, as well as service provider engagement, during a pediatric rehabilitation session from the perspective of an external observer.
ObjectiveTo translate and cross-culturally adapt the PRIME-O into Brazilian Portuguese and evaluate measurement properties.
MethodsThis study included translation, cross-cultural adaptation, back-translation, analysis of face validity, test-retest reliability, standard error of measurement, internal consistency, and Bland-Altman analysis of the PRIME-O Brazilian version. Fifty-one respondents included physical therapists (n = 47) and occupational therapists (n = 4) working in pediatric rehabilitation in Brazil.
ResultsTranslation and cross-cultural adaptation resulted in a Brazilian version of the PRIME-O that is easily understandable with local language appropriateness, semantically equivalent to the original version. In the face validity analysis, all PRIME-O items were considered useful, appropriate, and relevant for assessing engagement. Test-retest reliability ranged from 0.81 to 0.93 across all domains and total score. Cronbach's alpha ranged from 0.90 to 0.95 and the SEM ranged from 0.40 to 0.42 on average across domains with a value of 0.27 for the total test score. Bland-Altman plot analysis showed that most data points were within agreement limits, without proportional bias (p = 0.74).
ConclusionThe PRIME-O Brazilian version is a reliable and valid tool for measuring client and service provider engagement in pediatric rehabilitation sessions.
Engagement refers to the involvement of the client/family and the healthcare professional in the therapeutic process, and is considered a crucial component in the delivery of rehabilitation services.1 In pediatric rehabilitation, there is a growing interest in measuring the engagement of the child and their family during the provision of an intervention.2,3 Families are crucial agents of change, and their active participation in all stages of rehabilitation is essential.4 Building a therapeutic alliance and fostering a collaborative partnership among the child, family, and rehabilitation team enhances engagement in interventions.1 Engagement also motivates both the child and family to work towards their intervention goals beyond the therapy setting, whether at home or in the community, thereby contributing to achieving desired outcomes.5
Engagement is co-constructed by the service provider and the client within a complex system of psychosocial constructs, which include expectations, interpersonal relationships, receptivity, willingness, and self-efficacy of both the child and the family during therapy sessions.6 King et al.7 proposed a conceptual model for engagement in rehabilitation sessions, defining it as a multifaceted state of motivational commitment exhibited by the client and/or family during therapy sessions, divided into three components: affective, cognitive, and behavioral. The affective component pertains to the emotional involvement with the intervention and the trust relationship established with the therapist. The cognitive component involves beliefs about the necessity and effectiveness of the intervention being offered, which is demonstrated by their commitment to the therapeutic process. The behavioral component reflects the client's active participation during therapy sessions, including goal-setting and decision-making, as well as their confidence in their ability to perform activities.7
The components of engagement, while conceptually distinct, work together and concurrently influence the individual's progress in rehabilitation.3,7 Engagement changes throughout the therapeutic process, and in an ideal state of intervention, clients become highly motivated to achieve their goals.1,8 Characteristics of the client, the intervention, and the service provider are determining factors for optimizing engagement.5,8 Other aspects, unrelated to the intervention, can also influence the client's level of engagement, such as family context, daily routine, and availability to implement therapeutic activities at home.3 Measuring engagement is a challenge, especially in pediatric rehabilitation, because both children and families are considered clients.1,5
A systematic review of measures of parent engagement during therapeutic interventions found that all included studies measured only the behavioral component. This highlights the need for tools capable of capturing all components of engagement.3 The Pediatric Rehabilitation Intervention Measure of Engagement - Observation Version (PRIME-O) was developed with the aim of quantitatively assessing engagement during pediatric rehabilitation sessions, considering the affective, cognitive, and behavioral components of both the client — which may be the child, adolescent, or parent/caregiver — and the service provider. The PRIME-O is an observational measure with potential applications in research, clinical practice, and continuing education.5 It is a discriminative measure that captures differences in the development of engagement indicators, which are related to achieving meaningful outcomes in the rehabilitation process, assessing engagement as a co-constructed experience with a focus on the intervention as received.1,5 It can also be conceptualized as an educational tool, enabling service providers to critically reflect on their engagement practices and develop strategies to enhance their involvement.5 This standardized tool considers the perspective of an external observer analyzing verbal and non-verbal behaviors related to the mutual engagement between the service provider and the client.5
The original English version PRIME-O has easy applicability, demonstrating construct and content validity, as well as excellent inter-rater consensus.5 However, it has not yet been translated into Brazilian Portuguese. Measuring engagement within the context of rehabilitation allows for a deeper understanding of the complexities of the therapeutic process and facilitates the monitoring of changes in client involvement over time, particularly in rehabilitation programs with medium- and long-term goals, where engagement plays a key mediating role in achieving positive outcomes related to clinical change.1,5,8 Thus, the aim of this study was to translate and cross-culturally adapt the PRIME-O into Brazilian Portuguese and evaluate its face validity, test-retest reliability, standard error of measurement, and internal consistency.
MethodsThis is a methodological study guided by the recommendations of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)9 and the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN).10 The present study was approved by the Research Ethics Committee of the Universidade Federal de Minas Gerais (UFMG), Belo Horizonte, MG, Brazil (CAAE 54461321.2.0000.5149). All participants signed the Informed Consent Form.
Description of the PRIME-OThe PRIME-O is an observational measure with 10 items that assess the engagement of the client - whether a child, youth, or parent/caregiver - and the service provider during a rehabilitation session. The assessment is conducted by an observer external to the session and considers three domains: A) Client items: four items that consider the client's interest and enthusiasm, use of strength-based language, openness to what is being said/done, overall comfort, and confidence in engagement with the service provider; B) Service provider items: four items that consider the service provider's interest and enthusiasm, use of strength-based language, overall comfort, confidence in engaging the client, and listening/communication behaviors and; C) Client-provider interaction items: two items that consider the enthusiasm and collaborative nature of the interaction between the client and service provider.1
Each item on the scale is scored on a response scale with 0 to 4 points, where '0′ corresponds to "not at all," '4′ corresponds to "very much," and '2′ corresponds to "somewhat." The average score of the items is calculated to determine the engagement level within each domain and an overall engagement score. Higher values indicate higher levels of engagement.1 Additionally, the level of engagement in each affective, cognitive, and behavioral component can be identified by averaging specific items representing each component independently.5
Translation and cross-cultural adaptationThe translation and cross-cultural adaptation of the PRIME-O was carried out in accordance with the guidelines of Guillemin et al.11 and Beaton et al.,12 in line with the recommendations of the Scientific Advisory Committee of the Medical Outcomes Trust.13 The translation and cross-cultural adaptation process, formally authorized by the authors of the original version of the instrument, consisted of five stages. In the first stage, two translators who were proficient in English, experienced in pediatric rehabilitation, and native Brazilian Portuguese speakers independently translated the original instrument into Portuguese. The two translations were organized and synthesized into a final version, in which the two translators compared the obtained versions with the aim of producing a consensus version. In the third stage, the consensus version was sent to two independent bilingual translators, who were unaware of the original versions of the questionnaires and performed the back-translation of the instrument independently. In the fourth stage, a committee of experts (comprising three university professors from the Department of Physical Therapy at UFMG) compared the back-translated versions with the original version to ensure conceptual, idiomatic, semantic, and cultural adequacy of each item, resulting in a final back-translated version approved by the original authors of the instrument. In the fifth stage, the Brazilian version was tested by 10 professionals in the field of rehabilitation to ensure that users understood the content of PRIME-O.
Measurement propertiesFace validityTo determine face validity, 10 Brazilian physical therapists with at least 5 years of experience in pediatric rehabilitation were recruited as a convenience sample. The professionals responded to an online questionnaire with questions regarding the relevance of each item in the translated version of PRIME-O. Each item was scored using a three-point Likert scale (not relevant, partially relevant, or completely relevant). Items with 75 % or more positive responses (completely relevant and partially relevant) were considered useful and appropriate.
ReliabilityTo establish test-retest reliability, internal consistency, and standard error of measurement, 51 physical therapists and/or occupational therapists working in pediatric rehabilitation were recruited by convenience sampling. The sample size was based on the COSMIN10 reference parameters, which determine a minimum of 50 participants for evaluating the measurement properties of an instrument. Each participant analyzed two pediatric rehabilitation sessions, which included a family-centered approach and active participation of parents/caregivers during the sessions. Each session was conducted by a different physical therapist and involved a different family, being recorded in 45-minute videos. A researcher edited the video to obtain a 5-min segment, which was watched and analyzed by each participant for scoring the PRIME-O. The videos of the sessions were analyzed at two different times, with an interval of at least 15 days, to avoid memory bias.
Statistical analysisParticipant characteristics were analyzed using descriptive statistics (mean, standard deviation, frequency, and percentage). Test-retest reliability was determined using intraclass correlation coefficients (ICC2,1), considering 95 % confidence intervals (95 % CI).14 Values below 0.74 indicate poor to fair reliability, values between 0.75 and 0.89 indicate good reliability, and values above 0.90 indicate excellent reliability.15,16 Internal consistency was assessed using Cronbach's alpha to determine how well the items in each domain of the tool measured the same construct. Values between 0.70 and 0.90 indicate good correlation between items.15 The standard error of measurement (SEM) was calculated for each item and for the total score of each domain using the equation (SD × √[1 − ICC]). Bland-Altman analysis was performed to estimate the agreement between the measures (PRIME-O scores) in two different times, where a bias close to zero and small confidence intervals of the limits of agreement were interpreted also as a good indicator of reliability.17 The Statistical Package for Social Sciences was used for all analyses, and p-values <0.05 were considered statistically significant.
ResultsTranslation and cross-cultural adaptationThe adaptation of certain words in the instrument was carried out by the expert committee to enhance comprehension of the items by using terms commonly used in Brazilian Portuguese, always maintaining the original meaning. The response options in the instrument, with direct translation, included uncommon expressions that could hinder understanding, such as ‘moderately’ and ‘to a great extent,’ which were adapted to ‘more or less’ and ‘a lot.’ Literal translations of terms led to difficult-to-understand expressions; thus, some words were adapted for easier understanding: ‘open posture’ was replaced with ‘receptive posture,’ ‘configuration’ with ‘attitude,’ ‘interaction environment’ with ‘enthusiasm in interaction,’ and ‘connection’ with ‘overall relationship.’ The verb tense was changed from the infinitive to the present tense to give a sense of request/suggestion to the word, for example, changing ‘support’ to ‘supports’ and ‘encourage’ to ‘encourages.’ These changes were approved by the authors of the original version of the instrument (Table 1). The final version translated into Brazilian Portuguese can be seen at https://hollandbloorview.flintbox.com/technologies/9d2baf25-3d9a-4c5b-8dfc-a71b7e86bcbe.
Description of cross-cultural adaptations in the development of the PRIME-O Brazilian version.
The face validity of PRIME-O was analyzed by 10 pediatric physical therapists with a mean age of 42 years (±5.97) and a mean of 18 years (±6.03) of professional experience in teaching and research in pediatric rehabilitation. All items of the PRIME-O were considered useful and adequate for evaluating the engagement of the client and the service provider in a pediatric rehabilitation session, with positive classification (completely relevant and partially relevant) occurring for all items. The complete face validity data are described in Table 2.
Face validity of PRIME-O Brazilian version.
The main characteristics of the 51 professional participants for reliability measures are described in Table 3. Brazilian version of PRIME-O showed excellent test-retest reliability for both the total test score and the average score of the client-provider interaction items domain (Table 4). Both the individual items and the average score of client items domain showed good reliability. The service provider items domain showed good reliability for item 5 and the average score, while items 6, 7, and 8 showed fair reliability. The PRIME-O demonstrated good internal consistency, with Cronbach's alpha > 0.70 in all three scale domains (client items: 0.95; service provider items: 0.90; client-provider interaction items: 0.95). The SEM ranged from 0.43 to 0.65 for individual items and from 0.40 to 0.42 for the domain averages, with a value of 0.27 for the total test score. For the test items, the SEM ranged from 0.43 to 0.65 (Table 4). The Bland-Altman plot analysis showed that most data were distributed within the limits of agreement, without proportional bias (p = 0.74) (Fig. 1).
Participants’ characteristics.
Internal consistency and reliability of the Brazilian version of PRIME-O.
α, Cronbach's alpha; CI, Confidence Interval; ICC, Intraclass Correlation Coefficient; SEM, Standard Error of Measurement.
This study provides information on the translation and cross-cultural adaptation of the PRIME-O questionnaire to Brazilian Portuguese, following recommended international guidelines. This process yielded a version of the tool that is easy to understand and linguistically appropriate for the local context. The evaluation of measurement properties demonstrated appropriate face validity, indicating that the items in the Brazilian version are useful and suitable for assessing engagement in a pediatric rehabilitation session, confirming the tool's relevance to its intended purpose. PRIME-O showed good internal consistency of items and excellent test-retest reliability for the total score.
The translation of PRIME-O to Brazilian Portuguese raised some issues regarding the simplification of words and expressions for better understanding of the questions' meanings. For example, ‘open posture’ was replaced with ‘receptive posture,’ and ‘configuration’ with ‘attitude,’ enabling the evaluator to discern the construct being assessed and properly analyze engagement in its specific context within the session. The assessment of engagement is based on a recent conceptualization, requiring deeper exploration of its nuances and characteristics.1 Understanding engagement can enhance client/family-centered approaches, facilitating the identification of factors associated with goal achievement in pediatric rehabilitation.5
The Brazilian version of the PRIME-O demonstrated good consistency, indicating that the tool's domains are adequately correlated in measuring engagement. Test-retest reliability was excellent or good in all domains and most individual items.12,13 The exception was the individual analysis of the service provider items domain, which showed reasonable test-retest reliability. These items assess aspects related to the use of positive language that recognizes client strengths, the comfort and confidence of the service provider in engaging the client, and active listening behaviors associated with good communication skills.
In the original PRIME-O development study by King et al.5 service providers reported that in clinical practice, there are few experiential examples related to the use of strengths-based language and the collaborative nature of interactions. Service providers tend to focus more on the client's strategies than on their own or the client-provider relationship, which may explain the reasonable reliability in these aspects. In the present study, the evaluation of engagement during the intervention session was conducted by another professional through the observation of clippings from the video of pediatric rehabilitation sessions focusing on active family participation. The professional usually focuses almost entirely on client engagement in their clinical practice, making it less common to evaluate another professional's role as a relational contributor to engagement in rehabilitation. These factors may have influenced the observation of these items and consequently reduced the test-retest reliability values.
It is worth pointing out that the analysis of the SEM values for the Brazilian version of PRIME-O also indicated adequate reliability, as all values for both individual items and domains were below 0.85, demonstrating minimal dispersion of measurement errors in estimating true scores from observed scores.12 In the Bland-Altman analysis, most data were distributed within the limits of agreement, with minimal variability and no proportional bias.15
When developing the original version of PRIME-O,5 the measurement properties described in the present study were not evaluated, making it impossible to compare the values obtained, which can be considered a limitation of the present study. Furthermore, to date, no translated versions of PRIME-O into other languages have been found. Another possible limitation would be the analysis of 5-minute video clips that might not contain sufficient information to assess each of the PRIME-O items, making it important to assess the measure's applicability in complete pediatric rehabilitation sessions.
PRIME-O is based on the external observer's view of a pediatric rehabilitation session, observing behaviors related to the engagement characteristics of the service provider, the client (whether the child or the family), and the interaction between them.5 Therefore, one might question whether the assessor's knowledge of the concept and nuances of engagement, and their depth of understanding on this topic could potentially limit the thorough analysis of the session. To streamline the assessment process, the questionnaire already specifies which characteristics of the individual should be analyzed to score each item. For example, in the analysis of 'Interest and Enthusiasm,' for both the client and the service provider, aspects such as eye contact, receptive posture, sustained interest, energy, and enthusiasm should be observed for adequate analysis of engagement in this item, making the tool self-explanatory and intuitive to apply.
The PRIME-O is a short and easy-to-apply tool that measures engagement as a co-constructed experience focusing on the offered intervention. It can be applied via in-person sessions, telehealth, or recorded sessions.5 The items assess affective, cognitive, and behavioral aspects inherent to engagement in rehabilitation. It is an important tool for clinical practice, providing detailed information about both client and service provider engagement in rehabilitation and how this engagement changes throughout the intervention process. The tool provides a means to accurately identify observational signs of engagement, thereby facilitating the creation of a therapeutic environment that promotes involvement.5,6 Its applicability allows therapists to better understand the client's engagement in the intervention, facilitating the identification of needs and the implementation of targeted strategies.3,5 The PRIME-O is a valuable resource for understanding the complexity of the therapeutic process, modifying clinical outcomes and measuring the effectiveness of interventions. Engagement plays a crucial role as a mediator in the client-health professional dyad and can be seen as a facilitator in achieving the goals desired by the family.1,5
Moreover, the PRIME-O is particularly beneficial for students and professionals with limited clinical experience, as it clarifies and concretizes the relational aspects of practice. It helps educators support students' clinical reasoning and can be used to identify barriers and facilitators to client engagement.5 As a self-reflection tool, it enables professionals to document observations, generate follow-up actions, and deepen their understanding of engagement indicators.3,5 Additionally, service providers can use it to monitor engagement levels in clinics, identify areas for improvement, and promote family-centered services.
ConclusionThe Brazilian version of the PRIME-O is a reliable and valid tool for measuring client and service provider engagement in a pediatric rehabilitation session. Clinical applicability encompasses longitudinally measuring engagement throughout the implementation of an intervention and understanding how specific attitudes and interactions within the client-service provider dyad influence the attainment of significant clinical outcomes for the child and their family.
