A new study on SI effectiveness but measurement conundrums persist
Just a quick couple thoughts as I got a few emails asking me to comment on an article in the new AJOT. The study people are interested in is Effectiveness of sensory integration interventions in children with autism spectrum disorders: A pilot study. (linked for those of you who have access). The study found that both intervention groups demonstrated significant improvements toward goals on the Goal Attainment Scale, but the SI group demonstrated more significant improvement than the FM group. Also, the SI group displayed significantly fewer autistic mannerisms than the FM group as measured by a sub test of the Social Responsiveness Scale.
One of the nicest features of this study is that the researchers completed a series of fidelity measures on the interventions. This is a big step forward because so many of our research studies state that the intervention reflects sensory integration but this particular effort describes a solid fidelity test for each of the interventions provided (fine motor and sensory integration).
The authors state that "one of the purposes of this pilot study was to provide information to guide the development of future RCTs..." I believe that they met this objective well, particularly in consideration of the fidelity measures.
The researchers report that heterogeneity in their intervention groups may have impacted their results. This is probably quite a profound issue. Although we don't know the actual differences between individuals in the study, the tremendous variability in functional levels in ASDs is something that probably should be addressed in study design. The researchers took some positive steps to exclude people who have Asperger Syndrome and to restrict participation to Autism or PDDNOS. This is a good step, but I am not sure that it is an adequate step given the variability in this population.
There are a few design issues that I think need additional discussion. It is difficult to judge the relevance of the outcomes because there is not a non-intervention group. This makes it very difficult to factor out any Hawthorne-type effects. Similarly, the authors conclude that the study supports use of Goal Attainment Scales (GAS) for this type of research but I am not so sure about this. The challenges of GAS include possibly relevant confirmation or expectancy biases if the measurements are being made by the intervening therapists or the parents of the children in the study. I think that GAS can be strengthened if there is a way to more objectively measure progress based on those GAS measures by having independent assessment of results.
Use of the QNST-2 and the SPM are also appropriately identified as limiting because neither is designed or has been confirmed to be a valid pre-test post-test measure. I think that this is good reason to further narrow subject selection and find accepted tools that measure change across time instead of using this as justification to use GAS. That is easier said than done - undoubtedly - but it is just a thought.
The intervention provided in the study was quite intensive (three times per week for a six week period). In actual practice this kind of intensity is often not feasible (either given insurance restrictions or school district authorizations). This makes it a little difficult to know how this could or would generalize in a real-world setting.
Finally, the study is a little unclear about who was providing the SI intervention. The study explicitly states that a graduate student provided the FM intervention under supervision. It is unclear if the graduate students are certified or not, but given that they were being supervised it seems to indicate that they might not be certified. Given the relative lack of experience of an uncertified graduate student and the particular challenges of providing effective intervention to children who have ASDs, it might be important to control for the experience of the therapist between the two groups. It is not clear if graduate students also provided intervention to the SI group. This is a critical distinction and could have an important impact on progress made by the different groups.
I encourage everyone to read and discuss this study. It is a good step forward in our research regarding sensory integration and gives a lot of important information on how future studies might be designed.
References:
Pfeiffer, B.A., Koenig, K., et.al. (2011) Effectiveness of sensory integration interventions in children with autism spectrum disorders: A pilot study. American Journal of Occupational Therapy, 65, 76-85.
One of the nicest features of this study is that the researchers completed a series of fidelity measures on the interventions. This is a big step forward because so many of our research studies state that the intervention reflects sensory integration but this particular effort describes a solid fidelity test for each of the interventions provided (fine motor and sensory integration).
The authors state that "one of the purposes of this pilot study was to provide information to guide the development of future RCTs..." I believe that they met this objective well, particularly in consideration of the fidelity measures.
The researchers report that heterogeneity in their intervention groups may have impacted their results. This is probably quite a profound issue. Although we don't know the actual differences between individuals in the study, the tremendous variability in functional levels in ASDs is something that probably should be addressed in study design. The researchers took some positive steps to exclude people who have Asperger Syndrome and to restrict participation to Autism or PDDNOS. This is a good step, but I am not sure that it is an adequate step given the variability in this population.
There are a few design issues that I think need additional discussion. It is difficult to judge the relevance of the outcomes because there is not a non-intervention group. This makes it very difficult to factor out any Hawthorne-type effects. Similarly, the authors conclude that the study supports use of Goal Attainment Scales (GAS) for this type of research but I am not so sure about this. The challenges of GAS include possibly relevant confirmation or expectancy biases if the measurements are being made by the intervening therapists or the parents of the children in the study. I think that GAS can be strengthened if there is a way to more objectively measure progress based on those GAS measures by having independent assessment of results.
Use of the QNST-2 and the SPM are also appropriately identified as limiting because neither is designed or has been confirmed to be a valid pre-test post-test measure. I think that this is good reason to further narrow subject selection and find accepted tools that measure change across time instead of using this as justification to use GAS. That is easier said than done - undoubtedly - but it is just a thought.
The intervention provided in the study was quite intensive (three times per week for a six week period). In actual practice this kind of intensity is often not feasible (either given insurance restrictions or school district authorizations). This makes it a little difficult to know how this could or would generalize in a real-world setting.
Finally, the study is a little unclear about who was providing the SI intervention. The study explicitly states that a graduate student provided the FM intervention under supervision. It is unclear if the graduate students are certified or not, but given that they were being supervised it seems to indicate that they might not be certified. Given the relative lack of experience of an uncertified graduate student and the particular challenges of providing effective intervention to children who have ASDs, it might be important to control for the experience of the therapist between the two groups. It is not clear if graduate students also provided intervention to the SI group. This is a critical distinction and could have an important impact on progress made by the different groups.
I encourage everyone to read and discuss this study. It is a good step forward in our research regarding sensory integration and gives a lot of important information on how future studies might be designed.
References:
Pfeiffer, B.A., Koenig, K., et.al. (2011) Effectiveness of sensory integration interventions in children with autism spectrum disorders: A pilot study. American Journal of Occupational Therapy, 65, 76-85.
Comments