Just Recertified--Am I competent, good, outstanding . . . or not so much? Bob Wachter wants to know.

In a year-old post on The Health 2.0 Blog Bob Wachter points to the activities of commercial ventures including Healthgrades, Zagat and Google to begin rating doctors right along with your favorite city magazine. As is typical for his straightforward style, Dr. Wachter puts himself in the position of seeking information for his own care and concludes that he wants all the differentiating information he can get--including Board Certification ranking of some sort. He would prefer a doctor who scored at the 87th percentile to a doctor who scored at the 5th percentile. That sounds like it makes sense--doesn't it? Seems pretty sensible--a lot of face validity in that opinion--don't you think so, too?

Not so fast; I'm not so sure it works out so well for all doctors.

In the post Dr. Wachter discusses a presentation by Dr. Kevin Weiss, the president of the American Board of Medical Specialties made to the American Board of Internal Medicine (ABIM). Bob Wachter also reveals that he serves on the ABIM and Google's Healthcare Advisory Board; I served for nine years on the American Board of Emergency Medicine (ABEM) and I still read the periodic newsletters from the Executive Director there--I have some idea of what's going on in my own specialty certifying process. I also took and passed my recertification exam this year.

Our board, ABEM, uses criterion referenced examinations rather than norm referenced examinations. I'm not certain our specialty is still alone in that distinction, but at one time we were. While I'm not a psychometrician and I don't play one on TV, either, I've come to understand that our exam is pretty good at distinguishing a doctor who know 74% of the tested material from a doctor who knows 75% of the tested material. The latter will pass; the former will not. As I understand the nature of the exam and the scoring, while it is fair to say that the doctor who knows 100% of the tested material certainly knows more than the doctor who knows 75% of the test material, it's not at all clear that one knows a third more than the other or that the test can tell that the doctor at 100% knows more than a doctor scoring at 90%; it becomes even less certain as the differences become smaller; consequently, the American Board of Emergency Medicine probably can't put its diplomates on a percentile scale as Dr. Wachter suggests may be appropriate, at least probably not based on the testing approach in use today. Not to say that the testing couldn't be changed at some future time.

So Dr. Wachter's smell test notwithstanding and the people's desire for physician ratings very much still in evidence, it's not at all clear to me that the route to the goal is as direct as Dr. Wachter suggests.

Greater transparency in support of better decision-making for patients is a desirable, laudable goal. Reliable physician ratings is probably not coming soon, though city magazines, HealthGrades, Zagat and Google are either already or shortly to begin publishing their own ratings--user beware.