Patients with neck pain (NP) commonly display impairments in cervical movement control. Improvements in technology used to assess movement control allow extraction of various parameters that could enable more precise classification, but this has not yet been investigated.
ObjectiveThis study aimed to assess validity (classification performance of different machine learning models) of different parameters and difficulty levels of the Butterfly cervical movement control test.
MethodsSixty-five patients with NP and fifty asymptomatic controls performed the Butterfly test. Random forest, support vector machines and logistic regression classification methods were used to build models based on individual or multiple parameters across different difficulty levels. The area under the curve (AUC), classification accuracy, F1 score, precision, recall, specificity, information gain and Gini index were reported.
ResultsModels using amplitude accuracy, time-on-target or undershoot achieved the highest classification performance. However, models combining directional accuracy or all parameters of the Butterfly test at all difficulty levels yielded the most balanced probability of classification.
ConclusionResults of our study suggest that either directional accuracy or all parameters of the Butterfly test (calculated for all difficulty levels) should be used for classifying patients with NP to provide the most clinically valid tool.
The American Physical Therapy Association has established guidelines for managing neck pain disorders, conditions associated with substantial health decline across different patient subgroups.1 Among these, individuals with movement coordination impairments commonly exhibit sensorimotor deficits, such as altered proprioception and impaired eye–head–neck coordination, particularly in chronic cases.2–5 Consequently, assessing and targeting movement coordination is essential for effective rehabilitation in this subgroup.
Various approaches have been developed to measure head, neck, and eye movement control.5–9 These methods differ in the sensorimotor components they evaluate. While cervical position sense is well studied, movement sense has received less attention despite providing unique insights into cervical motor control. Movement sense tests require precise spatial and temporal regulation of head and neck motion, relying on proprioceptive feedback and vestibular–oculomotor integration. Numerous biomechanical parameters such as smoothness of movement, velocity, acceleration, deceleration, and amplitude accuracy have been used to characterize movement control.5,10,11 These parameters are associated with kinesiophobia12–14 and altered movement strategies,15,16 suggesting patient-specific adaptations.
Simple movement control tests consistently differentiate symptomatic from asymptomatic individuals.7,8,17–19 Advances in measurement technology have increased the number of extractable parameters, although their interrelationships remain unclear. Some studies have analyzed only a few parameters,20,21 whereas others have used large parameter sets.5,9,22,23 Majcen Rosker and Rosker24 emphasized that interpreting multiple parameters simultaneously may yield deeper insights into movement alterations and motor strategies.
Despite existing research, it remains uncertain which parameters best classify individuals with neck pain and whether combining parameters improves diagnostic accuracy. Smoothness of movement (e.g., jerk index) has been identified as an important discriminator,10 yet recent findings suggest that self-paced velocity profiles may show stronger discriminant validity.9,25 Additionally, amplitude accuracy (deviation of the head from a target position) can distinguish symptomatic from asymptomatic individuals during multi-velocity tracking tasks.26
A valid27 and reliable28 protocol, the Butterfly test (formerly Fly test),29 includes three increasingly complex target profiles with unpredictable velocity changes and trajectories. Prior studies have reported differences in measures such as time on target, overshoot, undershoot, and amplitude accuracy. Although earlier research focused solely on amplitude accuracy across difficulty levels,30 more recent work has examined multiple parameters.5 However, no study has evaluated the test’s classification accuracy or discriminant validity. Evidence from other cervical tests, such as the smooth pursuit neck torsion test, shows improved classification when multiple predictors are included.31 Determining which Butterfly test parameters best differentiate between groups may therefore reveal specific movement control deficits.
A clearer understanding of the sensitivity of these outcome measures is highly relevant to physiotherapy practice. A recent scoping review identified the development of new clinometric data and evaluation of diagnostic tools as key research priorities.32 Expanding knowledge of movement control test sensitivity, particularly through nonlinear analyses of multiple parameters, may improve the detection of head and neck impairments and inform more targeted rehabilitation.
Therefore, the aim of this study was to determine the discriminant validity of the Butterfly test using machine learning classification models, by evaluating sensitivity and specificity based on single parameters and combinations of multiple parameters across three levels of task difficulty.
MethodsSixty-five patients with chronic idiopathic neck pain and fifty age- and sex-matched asymptomatic controls were enrolled in the study (from November 2020 to May 2021). Patients were recruited from orthopedic outpatient clinics and asymptomatic controls via social media and advertisements at university campus. All participants were screened for eligibility via a telephone interview prior to participating in the study. Inclusion criteria for enrollment were age between 18 and 65 years, patients with neck pain were required to report a pain intensity of greater than 3 on visual analogue scale (VAS). Exclusion criteria were previous traumatic injuries to the head or neck, shoulder and/or arm pain within the past year, any neurological or vestibular disorders, type II diabetes and diagnosed psychiatric disorders. Furthermore, all participants were required to refrain from taking medication or alcohol for 24 h prior to the study. Participants had to read and sign a consent form and were free to withdraw at any time. The study was approved by the national medical ethics committee (number: 0120–47/2020/6) and was performed in accordance with the declaration of Helsinki. The study protocol was not registered.
Measurement proceduresPatients were required to mark pain intensity on a 10-cm VAS. Test of neck kinaesthesia using movement control test (the Butterfly test) was carried out by an experienced rater. During the Butterfly test, participants wore a head-mounted inertial motion unit (IMU) sensor and were instructed to accurately follow a dynamic unpredictable target on a computer screen (NeckSmart, NeckCare Holding ehf., Reykjavik, Iceland) (Fig. 1). Target movements were performed along three predefined paths of increasing difficulty (easy, medium, and difficult) in a random order, with each path repeated three times, independently. Target movement path characteristics and test duration were predefined by the NeckCare software (NeckSmart, NeckCare Holding ehf., Reykjavik, Iceland) (for more details please see the relevant reference27).
Data analysis
Accuracy of head and neck movements during the Butterfly test for each individual difficulty level (average of three repetitions) was analyzed using the NeckSmart software applying the following parameters: amplitude accuracy (AA) indicated by mean deviations of movement in millimeters and representing how far in average the subject was away from the target, time on target (ToT) expressed at the relative time spent on the target during each trial expressed as a percentage of total trial time, relative time spent behind the target (undershoot - U) expresses as percentage of total trial time, relative time spent in front of the target (overshoot - O) expresses as percentage of total trial time and smoothness of movement (SM) calculated as an ratio between integrals of quadratic sums of the third derivative of head position data and target position data. Parameters of ToT, undershoot and overshoot were commonly named directional accuracy (DA) as they indicate the direction of head and neck movement relative to the reference target.
Statistical analysisStatistical analysis was performed in Orange datamining software (Orange 3.38.0, University of Ljubljana, Ljubljana, Slovenia). Descriptive statistics were presented as group median and interquartile range.
Random forest, support vector machines, and logistic regression machine learning algorithms were used to build different models for classifying 65 patients and 50 asymptomatic controls into the two groups based on individual or combined ((i) using all parameters for individual difficulty level, (ii) using individual parameter at all difficulty levels, or (iii) using all DA parameters at all difficulty levels) classifiers. These datamining algorithms were chosen because they proved to be efficient with small sample sizes as small as in our study.33–36 The most efficient algorithm was chosen for final analysis (most consistent in producing highest model performance parameters described below). No data preprocessing was performed for the random forest algorithm because no data were missing. For the support vector machines and logistic regression algorithms, the data were normalized using z-values. In addition, hyperparameter tuning was performed using custom-written Python code to fine-tune each data mining algorithm to achieve the highest classification accuracy. Finally, a sigmoid calibration was applied to each classification approach to further reduce the possibility of overfitting. All models were cross-validated with 5 folds. For models containing more than three parameters simultaneously, the most significant parameters were selected by recursive feature elimination process using a custom written Python code.
The performance of each model was analyzed using the area under the receiver operating characteristics curve (AUC) and classification accuracy (CA). AUC was used as the primary metric to assess the model’s ability to discriminate between patients with neck pain and asymptomatic controls, with values closer to 1 indicating better performance.37 CA, defined as the proportion of correctly classified cases among all participants was also reported based on a fixed probability threshold of 0.05.37 In addition, precision (the proportion of correctly classified patients with neck pain among all participants predicted to have neck pain), recall or sensitivity (the proportion of correctly identified patients with neck pain among actual patients with neck pain included in the study), and specificity (the proportion of correctly identified asymptomatic controls among all actual asymptomatic controls included in the study) were analyzed. Finally, F1 score, representing the harmonic mean of precision and recall, was calculated to summarize the balance between these two metrics, with values closer to 1 indicating better combined performance.38Furthermore,95% confidence intervals were calculated for all metrics listed.
Contribution of individual classifier to the model performance was analyzed using information gain and differences between the two groups were analyzed using Gini index (ranging from “no difference” – O to “large difference” between groups - 1). In addition, the performance of the models was analyzed using the calibration plots where ratio between the actual group probability and predicted group probability were compared. The closer the plots are aligned with the diagonal, the better the tuning of the model. When the calibration plots lie above the diagonal line, they represent underconfidence of the model and overconfidence when they lie below the diagonal line of the plot. Finally, Receiver operating characteristic curve analysis was performed for each model using single parameter, and the optimal cutoff point was determined using Youden’s index, which provides the threshold with the best balance between sensitivity and specificity.39
ResultsCharacteristics of recruited patients with idiopathic neck pain and asymptomatic controls are presented in Table 1.
Characteristics of asymptomatic controls and patients with idiopathic neck pain.
| Patients with idiopathic neck pain (n = 65) | Asymptomatic controls (n = 50) | |
|---|---|---|
| Men/Women | 31/34 | 22/28 |
| Age (SD) | 44.2 ± 9.7 years | 42.9 ± 10.1 years |
| VAS (SD) | 4.3 ± 1.8 | |
| Pain duration (SD) | 11.8 ± 8.6 months |
SD – standard deviation; VAS – visual analogue scale.
Group medians and interquartile range for each parameter at each difficulty level of the Butterfly test are presented in Fig. 2. In general, patients with neck pain presented with higher values of AA, undershoot, overshoot and SM while lower values were observed for ToT as compared to asymptomatic controls.
Group medians and interquartile ranges for each parameter at each difficulty level of the Butterfly test. AA – Amplitude accuracy; ToT – time on target, U – undershoot; O – overshoot; SM – smoothness of movement; Easy – easy level of the Butterfly test; Medium – medium level of the Butterfly test; difficult – Difficult level of the Butterfly test.
The highest classification performance of single parameter models was observed when using the logistic regression method compared to random forest and support vector machines models (Supplementary Table 1), therefore further analysis was performed using the logistic regression method. The highest AUC, CA, F1 and specificity values were observed for AA and ToT, followed by undershoot. The lowest AUC, CA, F1 and specificity were observed for overshoot and SM. Butterfly test parameters calculated from the medium difficulty level showed the lowest classification performance. When individual parameters from all three difficulty levels were used simultaneously as classifiers, the AUC, CA, F1, and specificity tended to slightly increase.
The importance of individual parameter for classification models was analyzed using the information gain. AA at easy and difficult levels as well as undershoot at easy level presented with the highest information gain, followed by ToT at all three difficulty levels, undershoot at medium level and AA at medium level. Differences between the two groups were analyzed using the Gini index. The highest Gini index values were observed for the AA at difficult level and undershoot at easy level, closely followed by AA at easy level, ToT at easy, difficult and medium level (Supplementary Table 1).
The lowest performance of classification models based on single parameters or their combination at single difficulty level was observed for the overshoot and SM parameters indicated by lowest AUC, CA, F1 score, specificity and Gini index. Furthermore, these parameters showed the lowest information gain values.
Cutoff values for individual parameters, determined using Youden’s index, are presented in Supplementary Table 1. In general, these cutoff values varied with the increasing difficulty level of the movement control test. Specifically, the cutoff values for ToT and SM tended to decrease, while those for AA, U, and O increased as the test became more challenging.
Performance of models where combination of directional accuracy parameters or all parameters of the Butterfly test were combined at all three difficulty levels is presented in Table 2. Similar as in single parameter models, logistic regression provided the most efficient classification performance. Performance of models combining all parameters at a specific difficulty level was highest at easy and difficult level. High AUC, CA, F1 and specificity were observed for models using directional accuracy parameters at all difficulty levels and all parameters at all difficulty levels. In fact, the performance of these two models was comparable and slightly superior to models using AA and ToT as classifiers.
Classification performance of models based on combination of multiple parameters and all difficulty levels.
AUC – area under the receiver operating curve; CA – classification accuracy; F1 – F1 score indicating precision and recall; Precision – true positive rate divided by sum of true positive and false positive rate; Recall – true positive rate divided by sum of true positive and false negative rate; Specificity – true negative rate divided by sum of true negative and false positive rate; CI – 95% confidence interval; easy level – easy level of the Butterfly test; medium level – medium level of the Butterfly test; difficult level – difficult level of the Butterfly test; all levels – all levels of the Butterfly test combined; DA – directional accuracy parameters, combining time on target, undershoot and overshoot.
Performance of each model was additionally studied using the calibration plots, comparing actual and predicted group probability (Fig. 3). The models relying on individual parameters or individual parameters at all difficulty levels tended to be underconfident in classifying patients with neck pain (indicated by the calibration plot positioned above the diagonal line indicating perfectly calibrated model) and overconfident in classifying asymptomatic individuals (indicated by the calibration plot located below the diagonal line indicating perfectly calibrated model). A similar trend was observed for models where all parameters for individual difficulty level were combined (Fig. 4). The most balanced classification probability was observed for models using directional accuracy parameters or all Butterfly test parameters at all three difficulty levels (Fig. 5).
Calibration plots for models based on individual and combination of individual parameters at all difficulty levels. AA – Amplitude accuracy; ToT – time on target, U – undershoot; O – overshoot; SM – smoothness of movement; Easy – easy level of the Butterfly test; Medium – medium level of the Butterfly test; difficult – Difficult level of the Butterfly test; All difficulty levels – combination of individual parameter at all difficulty levels.
Classification results for models based on combination of parameters at all difficulty levels. All parameters – combination of all parameters at all difficulty levels of the Butterfly test; All DA parameters – combination of time-on-target, undershoot and overshoot at all difficulty levels of the Butterfly test.
The aim of this study was to analyze internal validity of a Butterfly movement control test for differentiating patients with neck pain from asymptomatic controls using machine learning-based algorithms incorporating different parameters. Different classification models were assessed, including those based on individual parameters and combinations of parameters, to identify the most sensitive and accurate model. The most efficient classification performance was achieved using combination of parameters, particularly when the following parameters were included simultaneously in the prediction model: (i) AA and ToT at all difficulty levels, (ii) all Butterfly test parameters at all difficulty levels or (iii) all directional accuracy parameters at all difficulty levels. These composite models outperformed those relying on individual parameters calculated at a single difficulty level. Among the individual parameters, AA and ToT, especially when measured at the easy and difficult levels of the Butterfly test emerged as the most sensitive for distinguishing between patients with neck pain and asymptomatic controls. In contrast, parameters such as SM and O demonstrated lower classification accuracy.
Accuracy of head and neck movement is commonly analyzed in cervical movement control tests,20,22,26 but studies findings remain inconclusive.29,30,40,41,44 In the study by de Zoete et al. ,30 the reported AA values from the Butterfly test ranged between 1.81 - 2.74 (easy level), 1.96 – 3.28 (medium level) and 2.24 – 3.71 (difficult level), while in the study by Kristjansson and Oddsdottir29 the values ranged between 1.54 – 3.30 (easy level), 1.19 – 4.90 (medium level) and 2.12 – 5.60 (difficult level) which is not consistent with the cutoff values reported in our study (cutoff value for easy level 2.230, medium level 4.440 and difficult level 6.735 respectively). These inconsistencies indicate that patients with neck pain represent a heterogenous group, experiencing larger variability of movement control impairments as indicated by a large variability in AA values. This discrepancy suggests that average values of AA and differences between patients with neck pain and asymptomatic controls reported in this study cannot be generalized for guiding clinicians in interpreting patients’ results (setting the cutoff values for determining altered AA).
In our study, AA as classifier demonstrated acceptable sensitivity and specificity, with the highest scores achieved at the difficult level. In addition, it proved to be an important classifier (higher information gain values) and showed the highest difference between the two groups as compared to other individual parameters (highest values of Gini index). However, the calibration plots indicate overconfidence or underconfidence (depending on the group being classified) of the models using AA (or in general all other single parameter classifiers) as classifiers and therefore making them less suitable for clinical use. This observation could be due to the fact that some asymptomatic control subjects (those with the lower actual group probability) present with inferior head and neck movement control similar to some patients with neck pain and were therefore classified as patients. The overconfidence of models using single classifiers was reduced when multiple parameters were included, allowing for more accurate classifications.
From all parameters analyzed during the Butterfly test, ToT showed highest AUC, CA, F1 score, and specificity, followed by the undershoot. ToT proved to be most sensitive at the easy and difficult levels of the test, however, undershoot achieved the highest F1 score and specificity, when it was used as classifier at easy and all difficulty levels. However, these two parameters provided a lower confidence model as presented in the calibration plots and are therefore less suitable.
High classification performance was achieved when all parameters of the Butterfly test or all directional accuracy parameters at all difficulty levels were combined as classifiers indicated by higher AUC, CA, F1 score, and specificity. Calibration plots revealed that the predictions of the two models were most balanced. This observation has two important implications. First, that classification model performance is most reliable when combining multiple parameters as classifiers and should present the preferred approach when assessing group differences or when used as a diagnostic tool. Secondly, dysfunctional head and neck movement deficits cannot be reliably identified using a single parameter.
The observations of this study are in line with observations by Majcen Rosker et al.,31 who investigated oculomotor performance in patients with neck pain and showed that classification models including multiple classifiers presented with superior performance. This could be partially explained by the fact that patients with neck pain constitute a heterogeneous group, where a wide range of individual parameter values could be expected. Therefore, incorporating multiple parameters could help to overcome such shortcomings.
An additional reason for the superior classification accuracy when applying multiple parameters could be presented by a recent study by Majcen Rosker and Rosker,24 in which three subcomponents of parameters were observed. For example, in their study, ToT, overshoot, AA, and SM were all part of the same components. The different component indicated that the difficulty level of the Butterfly test differentiates between the first two principal components. Moreover, undershoot presented the third component, suggesting that difficulty levels and the undershoot represent separate entities of motor control impairments in patients with neck pain. Therefore, it is reasonable to expect that the sensitivity and specificity (classification model performance) is highest when information from all components are included. Over relying on individual parameters of single difficulty levels therefore omits important information, which in contrast decreases F1 score and specificity of such models.
Above observations add important clinical insights. AA, ToT and undershoot were the most effective classifiers, suggesting slower and less accurate head and neck movements were observed in patients with neck pain. This conclusion is in line with another research, reporting slower and stiffer movements in patients with neck pain,42,45 which was confirmed by the results of this study. Furthermore, decreased accuracy could also be related to increased fear of movement as reported in other studies.12 However, additional research has to be performed in order to confirm these suggestions.
Overall, patients from our study seemed to demonstrate greater impairments than patients from other studies (e.g., higher AA values)29,30 indicating that the extent and differences between asymptomatic individuals and patients with neck pain when applying movement control test might not present with such differences as observed in our study, potentially suggesting that F1 score and specificity might differ, potentially showing lower values. Therefore, future studies should aim to identify potential subgroups of asymptomatic subjects and patients with neck pain based on the movement control performance and determine differences between patients adding information to better understand the above-described shortcomings of previous studies.
In addition to recruiting more homogeneous group of patients with neck pain, another limitation of our study was that only pain level, but no other symptoms were evaluated, such as central sensitization. Potentially, a better link could be established between symptoms and the extent of movement control impairment. Future studies should therefore add other symptoms related to neck pain disorders to classification models to see whether this could improve classification performance.
An important limitation of this study was that the group of asymptomatic controls were not evaluated for their ergonomic adaptations. Furthermore, their occupation, weekly working hours, and education were not taken into consideration when screening for potential risk factors, since these have been related to increased prevalence of neck pain.43,46 These postural adaptations might occur before the onset of neck pain but could affect proprioceptive sensory drive leading to impairment in movement control of the cervical spine. This might be the reason why some asymptomatic individuals participating in our study were misclassified.
Although the final models proposed as the most efficient in this study were adjusted to reduce overfitting (application of data preprocessing, cross-validation, simplification of the models by excluding irrelevant parameters and regularization techniques), this problem may persist due to the relatively small sample size and the lack of information on the reliability of the Butterfly test parameters. Therefore, future studies with a larger sample size should be conducted to confirm the results of this study. In addition, studies on the reliability of the Butterfly test are needed to assess the risk of possible data noise due to potentially lower repeatability of individual parameters.
In light of the identified limitations, future research should aim to develop more comprehensive classification models that integrate symptoms such as pain duration, central sensitization, oculomotor impairments, and cognitive and psychological alterations. Such models may offer a deeper understanding of the multifaceted nature of cervical movement control deficits and enable the identification of clinically meaningful patient subgroups. Stratifying patients based on these characteristics holds promise for optimizing rehabilitation strategies through more targeted and individualized movement control interventions. The effectiveness of these tailored interventions should be systematically evaluated to inform the development of evidence-based physical therapy treatments, assess their cost-effectiveness, and, most importantly, determine their impact on treatment responsiveness, as emphasized in a recent scoping review.32
ConclusionThe findings of this study indicate that the Butterfly test parameters demonstrate sufficient discriminant validity in distinguishing individuals with neck pain from asymptomatic controls. For optimal clinical applicability, either the DA or the full set of test parameters should be employed as classifiers. These results suggest that movement control deficits can be identified in patients with neck pain who may benefit from targeted movement and motor control interventions. Importantly, the absence of pain does not necessarily equate to intact motor control, underscoring the potential role of preventive strategies in asymptomatic individuals to mitigate the risk of developing neck pain.
The authors declare that they have no competing interest.
None to declare.
