Thursday, May 01, 2008

An attack on evidence-based assessment in occupational therapy

In the past I have blogged about evidence based practice so I don't really feel the need to revisit the topic at length. The use of evidence is not a passing fad but rather represents a real shift in the sophistication of our science. This shift was not just restricted to occupational therapy but occupational therapy was caught up in it. That was a good thing.

I am revisiting this tonight because I was visiting the WFOT website and I saw one of their banner ads. The ad was for Schoodles, and the ad said "Changing the way therapists around the world assess children." Kudos to the advertising people at Schoodles because I almost never click on banner ads but I never heard of this assessment tool and was intrigued by the claim.

Perhaps the Schoodles people will be sorry that I clicked because I don't really have anything flattering to say about their statements or philosophy.

This Schoodles product is antithetical to evidence based practice. Although I understand and appreciate the uses of qualitative and informal assessment methods, I was disturbed by the statements made on the website. Here is a sample of what I found that was particularly objectionable:

  1. "Standardized test scores can be valuable, but don't give enough information about classroom problems or starting points for intervention."
  2. "Most of the time occupational therapists don't need numbers and standardized scores to 'qualify' children for services or show continued need. Deciding which children could benefit from our services is really an art, not a science."
  3. "What do I do when teachers and parents want numbers? Give them numbers! You can give approximate age ranges for most of the skills on the tool. Parents typically understand terms such as '1-2 years below age level' much better than 1.5 standard deviations below the mean or 5th%ile. We have found that approximate age ranges satisfy most recipients of our information. You can also draw the focus away from numbers and instead stress classroom function, which is really what we need to look at for any student."

I suppose that their banner ad is correct, and that their assessment system would change the way that therapists assess children. In my opinion, it sets us back around 20 years or so in the sophistication of our assessments. That is a real change - unfortunately, in a negative direction.

The Schoodles approach is unfortunate. Even in the absence of norm-referenced assessments there are many wonderful criterion referenced tools that can be used. Why would anyone abandon these in favor of non-standardized methods???? Even worse, why be so flippant about using standardized methods? If someone asks for numbers the Schoodles people tell us to just 'Give them numbers!' Precision in measurement is not something that is important to the Schoodles people, apparently.

I do not know about the world that the Schoodles people operate in, but in my world school districts want to have norm referenced testing to determine eligibility for services. The same is true for early intervention and preschool services. Insurance companies also like to know where a child is functioning in relationship to a norm or established age criteria.

Most importantly, when parents want to know about the functioning of their children I don't blow smoke at them. Instead, I educate them about what statistics mean.

++++++++++++++++++++++++++++++++++++++

I wouldn't mind if this was marketed as 'an informal functional assessment tool to supplement your standardized measurements.' Instead I am disappointed with the grandiose claim of 'changing the way therapists evaluate children.' I am disappointed with the attack on evidence-based practice. I am disappointed with the flippant perspective on the value of standardized tools.

And IF WFOT has any control over the banner ads on their site, I am kind of wondering who is asleep at the wheel over there. In some ways, that is the most disappointing.

4 comments:

AFD said...

In WI, students don't "qualify" for OT based on numbers. It is up to the IEP team to determine if the student needs OT to benefit from their special education. I can't imagine the size of my caseload if it was just based on a standardized test score. Yikes!

Christopher Alterio said...

I think the larger point is that good standardized assessments should provide statistical evidence that relates to function in a real-world context.

There should be some correlation between the tools we are using and the validity of what they purport to measure. Hopefully, when we evaluate children we find that our results are a reasonable match to the actual needs of kids - of course subject to final team determination.

Marie said...

Hi Christopher,
I came across this posting.I apologize for my lateness in replying.
AFD is correct,Occupational Therapist is a related service in the US, meaning, we support special education services and therefore don't 'qualify' children.
We are about to work on our 4th update after 11 successful years of providing this tool to therapists and have thus far not received any negative feedback.
I am sorry we missed yours.
I actually appreciate your candid comments, it certainly gives us food for thought.
I want to point out that we did not pull our 'numbers' from the air.
We gathered information about each task from a number of researched based sources. They are included in our reference list.
We actually use statistical evidence to evaluate what we see and make our comments.
The tool also provides guidelines to use clinical judgement so often lacking when services are based on test scores only.
We wanted to do more when we put this together. We wanted to use and hone our observation skills.

We will be continuing to encourage the use of standardized tools, criterion referenced tools and skilled observation to best assess and serve children.

Marie
Schoodles Partner

Christopher Alterio said...

Hi Marie,

Thanks for commenting. The original blog post was written in 2008 - that is seven full years ago - so I wanted to provide readers with a proper time context in case things have changed. I have not looked at your website or products since that time and my comments were based on information available then. Now that it has come up, I will be interested to look at your current products. There has been a lot of conversation about evidence in the years since this original post so I am hopeful that some of that has filtered into your products. As I recall the products had an engaging appearance but I would have been more impressed with some evidence behind them. Thanks for stopping by.