CPOE reduces likelihood of error by nearly 50% [article]? I’m skeptical

There’s quite a bit of talk floating around the internet about a recent article in JAMIA that looks at reduction of medication errors in hospitals secondary to CPOE adoption (J Am Med Inform Assoc doi:10.1136/amiajnl-2012-001241). The article is available for free so I read through it last weekend. By the end I was looking at something that wasn’t all that impressive. The authors use a lot of sleight of hand, i.e. statistical models to tell a story about how CPOE “decreases the likelihood of error on that order by 48%”, which ultimately could potentially lead to a reduction in medication errors by approximately 12.5%”. That would be great, except that the entire thing is based on statistical models, assumptions, survey data and a great big meta-analysis.

Anyone that’s been in healthcare for more than 10 minutes and taken a basic statistics course knows about the use of a meta-analysis to tell a story. Basically you read the paper, cut their findings in half, reduce that outcome by 25% and then take the remainder with a grain of salt. I remember being at UCSF in pharmacy school and being scolded for using any paper whose results depended on a meta-analysis. My statistics professor at UCSF used to say “garbage in equals garbage out” when talking about their use.

Overall it’s a well written paper, but I never thought the medical community would latch onto it with such fervor. It’s become a CPOE supporters wet dream. I have to wonder how many of my colleagues took the time to read through it before Tweeting or re-Tweeting a link to it. That’s one of the problems with social media, people see something role across their feed and they pass it on without taking a deeper look at it first. Or that’s what I assume. I suppose there are those that simply don’t care that the article is suspect, as long as it supports the idea that CPOE is good. I believe CPOE is good. I believe it should be part of every healthcare system. But I also believe you have to be careful with the information you use to support the cause.

Ask yourself these questions before you read, or re-read the article:

General

• Where did the information come from?
• How old is the information?
• Is it a prospective randomized study with cross-over?
• Was direct observation used?

Specific

• Does it seem possible that CPOE alone can decrease the likelihood of a mistake by 48%, and decrease medication errors by 12.5%?
• What systems did they review?
• We all know how CPOE works, what mechanism would create such a drastic fall in errors? CPOE is akin to a pharmacy information system (PhIS). Do PhISs decrease errors in the pharmacy by 50%? Why not? What’s different?
• Are the results consistent with other CPOE studies? If not, what’s different?

Think about it people. Use common sense. Use your God given scientific knowledge. CPOE is good for healthcare, but be careful about using any ol’ thing to promote it. Make sure the sword you’re swinging isn’t made out of plastic.