Thursday, June 23, 2011

The lost art of evaluation in pediatric occupational therapy

This is a topic that is probably long overdue - it is something that I have observed in my geographic area for a long period of time. Based on conversations I have with therapists around the country I know that it is not exclusive to my area.

Increasing demands on therapist time, decreasing reimbursements, and dependence on a non-centralized workforce that is not subject to an intense quality improvement process has contributed to significant changes in how occupational therapy 'evaluations' are completed.

Our agency made attempts to impact this system while contracted to complete some evaluations, but 'contractor status' did not place us into a position to make broad system changes. Now it seems that the entire community is stuck into a cycle of low expectations based on long history. Unfortunately, there is no centralized service delivery system or 'top-down' method of assuring quality control - so I think the only way this can change is for the stakeholders to start increasing awareness of the problem and having a conversation.

Both therapists and parents should be interested in this issue. Therapists should be interested because I am calling the quality of our collective 'evaluation' work into question. Parents should be interested because they should know what to expect out of an occupational therapy evaluation for their children.

The American Occupational Therapy Association publishes standards of practice that outline some basics that should be part of all evaluations - but these standards are not very specific. Many evaluations that I see completed as a part of preschool and school-based practice do not even meet these basic standards. Currently, many evaluations that I am seeing can be outlined as follows:
  1. Very general statement regarding the child, including name, classroom, and very minimal if any history
  2. Long boilerplate descriptions of standardized tests that were used, followed by a very brief report of the child's performance on those tests
  3. A 'summary' section with minimal if any analysis and deferred decision making to the CPSE or CSE regarding eligibility for services.
I understand how 'evaluations' have devolved to this state - there are significant time demands and therapists do not have the time to complete lengthy evaluations. Also, there is no demand from CPSE or CSE for higher quality because the bottom line from the committee perspective is 'are children eligible or are they not eligible.' The committees are not in business to promote best practice; they are in business to determine eligibility. If a committee is receiving an evaluation report from a therapist it is generally making a de facto assumption that the evaluation being handed to them is appropriate and complete. Committees occasionally receive feedback about quality because some parents identify that the 'evaluations' are inadequate - and that causes external data (like a private evaluation) to be brought to the table.

External data in the form of a private occupational therapy evaluation causes a lot of consternation to committees, who don't understand 'what is wrong' with the evaluations completed within the system. That causes committee chairs and their committees to be puzzled at what is documented in a private assessment - and contributes to the general sense of 'that's not the way that we do things in schools' or even more commonly 'that is a clinical evaluation and not a school-based evaluation.' The most discouraging part of this problem to me is the engendered culture of low expectation and how practitioners create these myths about what constitutes acceptable practice.

The role of related service providers and their relative power within schools also contributes to the problem; this group of professionals is generally not in a position to take on the systems where they work and they are not always likely be on the front lines of change promotion within their systems.

Anyone who has been involved in this system for any length of time can identify with this issue.

So what can we do to change this?

I don't want to talk about 'best practice' because the term is overused and I also don't think it represents an intermediate step that we can reasonably take to improve. Let's talk about 'better practice.'

'Better practice' means taking a step forward from where we are now. It might not be best, but it is moving in that direction. From our current position, I propose the following for 'better practice' occupational therapy evaluations completed for CPSE and CSE:

  1. Background information including reason for referral, identification of medical issues, and lists of allergies and medications. Birth and developmental history need to be present, including history of CPSE or CSE involvement. The inclusion of developmental and medical history does not make an evaluation 'clinical.' This is basic information that is required to form a contextual understanding of the child's performance difficulties.
  2. Description of the child's ability to participate in the assessments. This also provides important contextual understanding of the results.
  3. A LISTING of the assessments used. If there is a value to 'explain' the tests for the parent audience then provide the parents with a separate sheet of paper with that information. The evaluation should never be 90% boilerplate explanation of what tests were administered.
  4. Direct performance observations AND performance on testing. Organize observations and test data into logical performance categories such as 'Physical skills,' 'Sensory skills,' 'Cognitive skills,' 'Regulatory skills,' and 'Social/emotional skills.'
  5. Apply this to actual function in their environment, including observations of how these performance attributes impact participation in personal care, learning, and play or socialization. Here it is likely that school based therapists will limit the 'environment of concern' to the school setting, which is appropriate.
  6. Summarize the findings, identifying areas of strength and areas of need. Form a summary opinion of what is happening with this child's life and make referrals for other services as needed.
  7. MAKE AN ACTUAL RECOMMENDATION! There is nothing wrong with giving your professional opinion. It is up to the committee to accept or reject your recommendations. That DOES NOT MEAN that you defer recommendations to the committee. This is where many committees fail - because it is absolutely fine for professionals to make recommendations and then for a committee as a whole to review those recommendations and decide what is most appropriate. For example, you may identify that a child's needs are so severe that you recommend OT three times a week. However, you may get to the committee meeting and find out that colleagues in PT, speech, and education made similar recommendations for their domains and in total it would be 'too much' for the child to tolerate in their day. The committee may then consider that a different level of service or an altered service delivery method is needed in consideration of ALL the data on the table. That is fine and is actually the STRENGTH of multidisciplinary planning.
  8. Follow up with the parents and teacher and talk to them on a regular basis so that you are not providing or documenting this service in a vacuum.
These are just some preliminary ideas. I am interested in feedback - again with the intention of promoting 'better practice.'

If we take some solid first steps, wouldn't it be nice to have a conversation about 'best practice' next?

7 comments:

Cheryl said...

sad when you find yourself just wishing everyone did a little more than the minimum required. Definitely been there.

adam smith said...

It can be recovered by doing proper OT. Its not that you do a little less or more. it should be done properly to overcome the problem.

Lee Rhodus, OTR/L said...

I can certainly see the need for "better practice" in the area of school based OT Evaluations. You've raised some very good points with the schools cse/cpse getting quite ruffled when the mere mention of an outside evaluation is proposed, usually by the parent or parent advocate. These days especially, everyone is expected to do more with less....less time, money, resourses....but we as OT's are not in school based practice for the money so we might as well do everything we can to ensure the best for our students, including performing adequate, through evaluations. What youve outlined as general improvements seem quite reasonable and should already be in most evaluations. I enjoyed reading some of your blogs and if I had more time I would respond more frequently. I'll be referring some of my staff to your blog as I've found it refreshing to hear your opinion on some very good topics. Thanks for posting.

Lee Rhodus, OTR/L

Anonymous said...

Initially I was interested because I am a true advocate for providing the HIGHEST quality of care, but I am extremely discouraged after reading the segment "about this blog's author." Unfortunately he presents as egotistical intelectual, simply seeking to initiate some notoriaty to achieve fame. I have the same degree as "said author" and I believe there is a way to impact change in the system, WITHOUT flaunting intellegence or degree.

Comment moderation has been enabled according to the blog author. I am completely aware that you (the blog author) will not submit my comment as written. Knowing this, I hope something I wrote will reach YOU. Since it OBVIOUSLY will not reach ANYONE else.

Chris said...

Ah, the professionalism of anonymous criticism. It is hard to know what people like this are really responding to. The venom overshadows any point they might be wanting to make.

Anyway, now that I have published your comments, how about you come back and put your name to what you wrote?

I want you to know that I think your comments are so fantastic that I have memorialized them in my blog review post at http://abctherapeutics.blogspot.com/2008/08/abc-therapeutics-occupational-therapy.html

You have reached an audience and your comments are published. Is there something more you would like to contribute? Something constructive? Or will you hide and remain anonymous?

Unknown said...

Dear Chris,

Thank you for writing this. I am a huge advocate of completing appropriate evaluations – because it validates why you are treating that child – you take baseline data, treat, re-evaluate that child with the same assessment, and (hopefully), you find improvement. Here is some background regarding the school that I work in – all of the children have a visual diagnosis, in addition to many medically complex diagnoses. None of the assessments available are appropriate for the population that I work in, so, based on research, the occupational therapists have developed a comprehensive evaluation that is completed by the end of the school year in combination with our clinical observations. Do you know of any published evaluations that accommodate for children of such medical complexity? What suggestions do you have for this type of setting? I feel that we try our best, but I wonder if it could be better. Thank you so much for your time.

Cortney

Christopher Alterio said...

Hi Cortney,

Trying to complete assessments on children who have visual impairments is extremely difficult. I am not aware of standardized materials that are appropriate for this population, and so I generally use curriculum based assessments and even then I comment that the assessments are not developed for children with visual difficulties and results must be interpreted with caution.

I suspect that the local assessment created by your team of expert therapists who have experience with your population is better than what anyone else could come up with! Otherwise, using generic developmental scales and trying to adapt materials as much as is needed is probably the best you could do.

Please keep in touch and let me know if you come across any instruments that are appropriate for your population.

Chris