A response to Hinojosa's "The Evidence-based paradox."
Jim Hinojosa (2013) wrote an interesting essay in the recent AJOT entitled "The evidence based paradox" which was published in "The Issue Is..." section. I believe that this is an important article to read and discuss.
I was a little concerned when I was reading the article because although he took some rather specific steps to tread cautiously and he did not go so far as an outright rejection of an evidence-based model there is still danger in propelling our thinking backward when you express the kinds of concerns that he expressed.
Hinojosa outlines barriers to EBP including possible inapplicability of the established levels of evidence given the important qualitative and individualized nature of OT practice. He correctly references the revised model that Tomlin and Borgetto (2011) presented that has already addressed this concern. Tomlin and Borgetto (2011) proposed a model of a 'Research Pyramid' that includes and expresses a value for qualitative inquiry, so I am not sure why this is still identified as a barrier by Hinojosa.
Hinojosa also criticizes systematic reviews for the potential of individualized bias and inclusion criteria errors but that is kind of like shooting fish in a barrel. Of course there will be the potential for bias in nearly every endeavor; the potential should not preclude adoption of methods that are otherwise sound as long as we maintain the ability to study and examine and even criticize the reviews! The point here is that even though there can be potential for 'problems' we should not take that as an invitation or opportunity to reject the method.
Evidence based practice is a standard and an objective - not an absolute. The reason why we know this is precisely related to the creation of the standards which includes reference to 'levels of evidence.' We can engage in academic conversations about whether or not the standards apply to the way that we practice but I think a more important method for framing the question is 'What happens when we step too far away from evidence-based models?'
Since I am a street level practitioner I would like to outline some things that happen when we step away from evidence. I see these things on a daily basis in my practice. Consider this:
When we step away from evidence based practice we are free to ignore good research and the efforts of many scientists who are attempting to better inform our practice strategies. That leaves us free to reject the fact that some interventions are simply not supported. What is the practical application of this? Well, if we reject evidence we are liberated to use treatment strategies that are not effective and we return to a lower point of theory driven decision making. Look no further than the state of pediatric practice to see this in action: we had a theory about a 'brushing' protocol that was sold like snake oil for over twenty-five years to thousands upon thousands of practitioners who paid money to learn the methods. Those thousands upon thousands of practitioners went out into the world and brushed an uncounted number of children - and justified their intervention based on what they learned from experts who were applying their 'theories.' They also brushed those uncounted children with the promised notion that 'research is coming!!!!' which was the mantra chanted in those continuing education courses - which I remember specifically because I attended one of the first ones back in 1989. Well fast forward to 2013 and in a retrospective analysis there has been an ENTIRE INDUSTRY developed around this intervention. Millions of dollars have been spent on continuing education and buying the 'right' brushes. It is impossible to calculate the dollar figure impact of how many sessions have been billed for this treatment. Now, what do we have to show for this treatment that is based on theory? We have absolutely NO EVIDENCE that it is even effective. In fact, it is not even referenced in our most basic entry level pediatric text books. I didn't edit those textbooks so I don't know what the decisions were related to why this extraordinarily mythical intervention was not included, but I suspect it had something to do with the fact that there is absolutely NO EVIDENCE that this is effective, and in fact those studies that have been done essentially show that there are no consistent effects that can cause us to have confidence in brushing with a 'particular method' and using a 'particular brush.'
Dr. Hinojosa is an academic and I am sure he must have contact with students who go out into the field and practice occupational therapy. What is an academic response to a student who looks in the basic pediatric textbooks and can't even find a reference to a brushing intervention that is so popularized by clinicians? Do we tell students that it is 'OK' to brush children based on our theories, even though our theories have not been supported by research - YET? EVER? MAYBE SOMEDAY???
HOW LONG DO WE WAIT??? Is 25 years long enough to wait before a clinician 'gives up' and assumes that this is snake oil and that research is actually NOT coming???
Here is another way to conduct the analysis in step with concerns that our practice is directed at an individual level and is therefore difficult to measure quantitatively. What I really want Dr. Hinojosa to reconsider is the impact of this on a family - a topic that he has studied (Hinojosa, 1990) in an article that I rather enjoyed. I evaluated a three year old child this week who has multiple developmental delays related to prematurity. Among many other problems, this child can not hold a toy because any time any object or person touches her hands she pulls them away. The parent WANTS the child to hold a toy and WANTS the child to play and not have this difficulty. At that moment I was confronted with a practice dilemma: do I practice by evidence or do I practice by mythology? What do I recommend??? I know the theory and am no clinical slouch. I understand Melczak and Wall (1965), I understand the animal research on deep pressure and calming (Grandin, 1992), and I also understand the theory of behavioral conditioning or placebo if that is what is in play, and I understand our mythic interventions that include brushing but that have no direct research support after 25 years.
So what are we supposed to do here, continue with mythic-based practice and subject the public to treatment strategies that any science-oriented thinker has long ago rejected? More specifically, with this child lying in the bean bag chair in front of me while the parent expresses her sadness that the child can't even hold onto a toy and play, what am I supposed to do?
Well at that moment, even though I can't find the reasons in my head to justify the use of brushing, I almost wanted to say the words to the family: 'Well you can TRY this. We don't really have any supporting evidence but some people find that it is helpful.' I thought of the study done on weighted vests that showed even though there was no change in the child at least it made the parents feel good (Stephenson & Carter, 2009).
Then the parent brought me back to reality. Even before I could say anything she said, "Please don't ask us to try brushing. Every therapist we have ever seen wants us to brush this child and I can't tell you how many times we have tried it. Some of them even blamed us for it not working because they said we weren't following the protocol closely enough."
If I am using ANY kind of evidence, even the qualitative kind, shouldn't I at least remember that parents 'know' more than therapists when it comes to this topic? Shouldn't I remember that when considering adherence to a brushing home program that a qualitative study indicated that parents are concerned with efficacy and response to the intervention while therapists were more concerned with whether or not the family had the time and was actually following the protocol (Segal & Beyer, 2006)?
++++
I don't believe that there is any paradox or even dilemma. In consideration of quantitative and qualitative evidence, if that even matters, is that the evidence seems to be useful for guiding our practice. Or informing practice, at least, which Hinojosa seems to support.
I just want him to consider that publication of an opinion piece like this can lead to some practitioners having some sense of being justified in resorting to a 'theory-based' methodology and rejecting calls for scientific thinking. Theory-based methodologies have their time and place - but we need to be careful that we are not feeding pseudoscientific models just because someone can dream up a proposed theoretical justification for something. That is happening a lot, particularly in pediatric occupational therapy practice, and there is not enough discussion about the problem.
Questioning the value of evidence based practice has the potential to be a giant step backward, and in total, I think the Tomlin and Borgetto model (2011) already addressed every concern that Hinojosa raised in his essay.
References:
Grandin, T. (1992). Calming effects of deep touch pressure in patients with autistic disorder, college students, and animals. Journal of Child and Adolescent Psychopharmacology, 2(1), 63–72.
Hinojosa, J. (1990). How mothers of preschool children with cerebral palsy perceive occupational and physical therapists and their influence on family life. Occupational Therapy
Journal of Research, 10, 144–162.
Hinojosa, J. (2013). The issue is... The evidence-based paradox. American Journal of Occupational Therapy, 67, e18-e23.
Melzack, R., & Wall, P.D. (1965). Pain mechanisms: a new theory. Science, 150, 971–979 .
Segal, R., & Beyer, C. (2006). Integration and application of a home treatment program: A study of parents and occupational therapists. American Journal of Occupational Therapy, 60, 500–
510.
Stephenson, J., & Carter, M. (2009). The use of weighted vests with children with autism spectrum disorders and other disabilities. Journal Of Autism And Developmental Disorders, 39(1), 105-114.
Tomlin, G. & Borgetto, B. (2011). Research pyramid: A new evidence-based practice
model for occupational therapy. American Journal of Occupational Therapy, 65(2), 189-196.
I was a little concerned when I was reading the article because although he took some rather specific steps to tread cautiously and he did not go so far as an outright rejection of an evidence-based model there is still danger in propelling our thinking backward when you express the kinds of concerns that he expressed.
Hinojosa outlines barriers to EBP including possible inapplicability of the established levels of evidence given the important qualitative and individualized nature of OT practice. He correctly references the revised model that Tomlin and Borgetto (2011) presented that has already addressed this concern. Tomlin and Borgetto (2011) proposed a model of a 'Research Pyramid' that includes and expresses a value for qualitative inquiry, so I am not sure why this is still identified as a barrier by Hinojosa.
Hinojosa also criticizes systematic reviews for the potential of individualized bias and inclusion criteria errors but that is kind of like shooting fish in a barrel. Of course there will be the potential for bias in nearly every endeavor; the potential should not preclude adoption of methods that are otherwise sound as long as we maintain the ability to study and examine and even criticize the reviews! The point here is that even though there can be potential for 'problems' we should not take that as an invitation or opportunity to reject the method.
Evidence based practice is a standard and an objective - not an absolute. The reason why we know this is precisely related to the creation of the standards which includes reference to 'levels of evidence.' We can engage in academic conversations about whether or not the standards apply to the way that we practice but I think a more important method for framing the question is 'What happens when we step too far away from evidence-based models?'
Since I am a street level practitioner I would like to outline some things that happen when we step away from evidence. I see these things on a daily basis in my practice. Consider this:
When we step away from evidence based practice we are free to ignore good research and the efforts of many scientists who are attempting to better inform our practice strategies. That leaves us free to reject the fact that some interventions are simply not supported. What is the practical application of this? Well, if we reject evidence we are liberated to use treatment strategies that are not effective and we return to a lower point of theory driven decision making. Look no further than the state of pediatric practice to see this in action: we had a theory about a 'brushing' protocol that was sold like snake oil for over twenty-five years to thousands upon thousands of practitioners who paid money to learn the methods. Those thousands upon thousands of practitioners went out into the world and brushed an uncounted number of children - and justified their intervention based on what they learned from experts who were applying their 'theories.' They also brushed those uncounted children with the promised notion that 'research is coming!!!!' which was the mantra chanted in those continuing education courses - which I remember specifically because I attended one of the first ones back in 1989. Well fast forward to 2013 and in a retrospective analysis there has been an ENTIRE INDUSTRY developed around this intervention. Millions of dollars have been spent on continuing education and buying the 'right' brushes. It is impossible to calculate the dollar figure impact of how many sessions have been billed for this treatment. Now, what do we have to show for this treatment that is based on theory? We have absolutely NO EVIDENCE that it is even effective. In fact, it is not even referenced in our most basic entry level pediatric text books. I didn't edit those textbooks so I don't know what the decisions were related to why this extraordinarily mythical intervention was not included, but I suspect it had something to do with the fact that there is absolutely NO EVIDENCE that this is effective, and in fact those studies that have been done essentially show that there are no consistent effects that can cause us to have confidence in brushing with a 'particular method' and using a 'particular brush.'
Dr. Hinojosa is an academic and I am sure he must have contact with students who go out into the field and practice occupational therapy. What is an academic response to a student who looks in the basic pediatric textbooks and can't even find a reference to a brushing intervention that is so popularized by clinicians? Do we tell students that it is 'OK' to brush children based on our theories, even though our theories have not been supported by research - YET? EVER? MAYBE SOMEDAY???
HOW LONG DO WE WAIT??? Is 25 years long enough to wait before a clinician 'gives up' and assumes that this is snake oil and that research is actually NOT coming???
Here is another way to conduct the analysis in step with concerns that our practice is directed at an individual level and is therefore difficult to measure quantitatively. What I really want Dr. Hinojosa to reconsider is the impact of this on a family - a topic that he has studied (Hinojosa, 1990) in an article that I rather enjoyed. I evaluated a three year old child this week who has multiple developmental delays related to prematurity. Among many other problems, this child can not hold a toy because any time any object or person touches her hands she pulls them away. The parent WANTS the child to hold a toy and WANTS the child to play and not have this difficulty. At that moment I was confronted with a practice dilemma: do I practice by evidence or do I practice by mythology? What do I recommend??? I know the theory and am no clinical slouch. I understand Melczak and Wall (1965), I understand the animal research on deep pressure and calming (Grandin, 1992), and I also understand the theory of behavioral conditioning or placebo if that is what is in play, and I understand our mythic interventions that include brushing but that have no direct research support after 25 years.
So what are we supposed to do here, continue with mythic-based practice and subject the public to treatment strategies that any science-oriented thinker has long ago rejected? More specifically, with this child lying in the bean bag chair in front of me while the parent expresses her sadness that the child can't even hold onto a toy and play, what am I supposed to do?
Well at that moment, even though I can't find the reasons in my head to justify the use of brushing, I almost wanted to say the words to the family: 'Well you can TRY this. We don't really have any supporting evidence but some people find that it is helpful.' I thought of the study done on weighted vests that showed even though there was no change in the child at least it made the parents feel good (Stephenson & Carter, 2009).
Then the parent brought me back to reality. Even before I could say anything she said, "Please don't ask us to try brushing. Every therapist we have ever seen wants us to brush this child and I can't tell you how many times we have tried it. Some of them even blamed us for it not working because they said we weren't following the protocol closely enough."
If I am using ANY kind of evidence, even the qualitative kind, shouldn't I at least remember that parents 'know' more than therapists when it comes to this topic? Shouldn't I remember that when considering adherence to a brushing home program that a qualitative study indicated that parents are concerned with efficacy and response to the intervention while therapists were more concerned with whether or not the family had the time and was actually following the protocol (Segal & Beyer, 2006)?
++++
I don't believe that there is any paradox or even dilemma. In consideration of quantitative and qualitative evidence, if that even matters, is that the evidence seems to be useful for guiding our practice. Or informing practice, at least, which Hinojosa seems to support.
I just want him to consider that publication of an opinion piece like this can lead to some practitioners having some sense of being justified in resorting to a 'theory-based' methodology and rejecting calls for scientific thinking. Theory-based methodologies have their time and place - but we need to be careful that we are not feeding pseudoscientific models just because someone can dream up a proposed theoretical justification for something. That is happening a lot, particularly in pediatric occupational therapy practice, and there is not enough discussion about the problem.
Questioning the value of evidence based practice has the potential to be a giant step backward, and in total, I think the Tomlin and Borgetto model (2011) already addressed every concern that Hinojosa raised in his essay.
References:
Grandin, T. (1992). Calming effects of deep touch pressure in patients with autistic disorder, college students, and animals. Journal of Child and Adolescent Psychopharmacology, 2(1), 63–72.
Hinojosa, J. (1990). How mothers of preschool children with cerebral palsy perceive occupational and physical therapists and their influence on family life. Occupational Therapy
Journal of Research, 10, 144–162.
Hinojosa, J. (2013). The issue is... The evidence-based paradox. American Journal of Occupational Therapy, 67, e18-e23.
Melzack, R., & Wall, P.D. (1965). Pain mechanisms: a new theory. Science, 150, 971–979 .
Segal, R., & Beyer, C. (2006). Integration and application of a home treatment program: A study of parents and occupational therapists. American Journal of Occupational Therapy, 60, 500–
510.
Stephenson, J., & Carter, M. (2009). The use of weighted vests with children with autism spectrum disorders and other disabilities. Journal Of Autism And Developmental Disorders, 39(1), 105-114.
Tomlin, G. & Borgetto, B. (2011). Research pyramid: A new evidence-based practice
model for occupational therapy. American Journal of Occupational Therapy, 65(2), 189-196.
Comments
Every article I have read regarding SI end is " further research is needed" yet OTs continue to push this treatment as a cure all for everything.
I too, based on one of your previous postings regarding SI and its lack of evidence, have stated to parents "there is no evidence, but you could try...if you have the resources". Now that I think of it, this still gives desperate parents a false hope of being able to "fix" something and putting on the hampster wheel of always waiting for something else that will work instead of focusing onthe present.
Of course the problem with the sensory research is the fidelity problem, which I have documented rather extensively. Here we get deep in the weeds about what our knowledge about sensory problems indicates - for example, does it indicate that there indeed are problems that require attention or does it provide information on what kinds of intervention the research supports? Making this distinction is at the core of our current need on this issue in my opinion. We have not adequately distinguished 'sensory' from 'other' problems and we have been too quick to jump in with recommendations for treatment of a problem that we are incompletely (and sometimes inaccurately) defining.
I would to thank you for speaking out on this issue. I think you raise some very important points, and from my view of the OT world our profession is coming dangerously close to losing the respect of the medical community.
As an OT, who has practiced in the school systems for 10 years in 5 different states, I can confirm that this blatant lack of regard for evidence and rigor in our field is stunningly widespread. I am currently in a position as a contract therapist who provides distance service to rural school districts in Oregon and Washington. This gives me a glimpse into the climate of the isolated Occupational Therapists working in these environments; my current contract has me working in a region program that has accepted Bal-A-Vis-X as the corner stone of their intervention plans. Not only are these therapists wrongly applying a pseudoscience based perceptual motor program to a broad range of students (from the student with a mild case of ADHD to a child with severe CP who has limited hand functioning) with a subjective treatment goal (improve attention) but the most frightening thing is they appear to not be able to recognize pseudoscience. I have seen this inability played out over and over again in each district and state that I work in. Therapists seem to take the words of a continuing education provider as law (and you are being nice to say that presenters promise research, I’ve been to several conferences where they offer “we all know the miracles of brushing” as the evidence to support their claims), and typically I can trace which conferences have come within 100 miles of a district by what interventions are being whitewashed across therapist’s caseloads. Mary Kwar and her astronaut training is making a killing in Skagit Valley, Washington; Luisa Silva’s Quigong Massage for Autism is “curing” kids with “Autism” in Eastern Oregon, Paula Aquilla’s sensory diets are “prescribed” for every kid who flaps his hands in Clackamas, Oregon, the list goes on and on.
My point is this, as therapists we must demand better evidence and accountability from the “leaders” in our field; Dr. Clark, while being a champion of OT, should be providing the voice of reason in the sensory integration debate not throwing out conspiracy theories, Dr. Schaaf and Dr. Miller should be reaching out to the leaders of the ABA field to work together and learn from each other, AOTA should require continuing education course providers to provide solid research behind their claims, education should be provided to therapists regarding how to document intervention effectiveness and evaluate the claims of those selling products, evidence to support effective interventions should be shouted from the roof tops not only reported in the confines of the cumbersome AOTA website- I could go on, but that seems like a good start.
My second point, and I think the most important one is, the responsibility does not end with the leaders in the OT world, as street level therapists we must demand more from our colleagues. It should not be acceptable to receive a file from a therapist with limited documentation, sparsely informative evaluations and exclusive use of unproven interventions. No longer can we accept whining about caseload size and driving times as legitimate reasons to be ineffective therapists. We must expect that our colleagues are adhering to the code of ethics published by the AOTA that expressly states that evidence based interventions be given preference over those with limited or no evidence behind them.
Again, Christopher, I would like to commend you for blogging about these issues; it’s nice to know that I’m not the only outlier in our profession. I do, however, find it disturbing that these “dissenting” opinions are found only in the blog format and are not seen in the formal publications of our professions. I wonder if you have any thoughts or suggestions towards how to initiate these changes within the field?
Best,
Gretchen