Prodded by Congress, Medicare will tie more of its compensation for doctors and hospitals to the quality of their care. And who, you might ask, could be against such pay for performance–besides incompetent providers trying to preserve their reimbursements?  Doesn’t it make sense to pay docs and hospitals for improving the health of their patients rather than for the volume of tests, therapies, and other services they provide?

Sure it does. But there is one problem: We are not very good at measuring whether a doctor really improved the health of her patient. So Medicare uses proxies for quality—hundreds of benchmarks that look at the process of care, not the actual outcomes. And increasingly, health policy experts are pushing back. One of the most outspoken is my Urban Institute colleague Bob Berenson.

Make no mistake: Berenson, who is a physician as well as a policy expert, is a strong advocate for better quality health care. But he, and others, worry that poor performance measurement not only won’t improve care, it could make it worse. This is what he wrote in a 2013 New England Journal of Medicine essay (along with coauthor Deborah Kaye of Johns Hopkins):

“The Centers for Medicare and Medicaid Services (CMS)…cannot accurately measure any physician’s overall value, now or in the foreseeable future….Instead of helping to establish a central role for performance measurement in holding providers more accountable for the care they provide…this policy could undermine the quest for high value health care.”

As Bob notes, it is not for lack of trying. CMS has been struggling for years to create good performance measurement benchmarks. But it is very hard (Bob and others would say impossible). For one thing, outcomes themselves are not easy to measure. An 80-year-old goes to the doctor with back pain. What is the best outcome? No pain? That’s probably impossible to achieve with even the highest quality care. Less pain? Maybe. But what does that mean and how do you measure it from patient to patient?

Then there is the matter of adjusting those scores for the severity of the disease and the social and economic status of the patient. This matters because low-income patients often struggle to manage their follow-up care or may be unable to afford medications. Such “risk adjustment” is even harder to do with older adults with multiple chronic conditions.

But Bob goes beyond saying that pay for performance doesn’t work very well. He says it actually can undermine the very goal of quality healthcare it is trying to improve. What’s that about?

In their essay, Berenson and Kaye say that one common definition of a physician’s competence is “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served.”

That sounds like the kind of doc I’d like to see. But think of it: Almost none of those qualities can be counted. And if they can’t be measured, they won’t be included in the benchmarks that drive performance-based pay.

That’s where the model runs up on the rocks. If Medicare tells doctors and hospitals that they must meet certain goals in order to avoid financial penalties, the providers will, more than likely, work to meet those goals. But the effort it takes to do so may perversely distract them for the very qualities that make them good providers—those unmeasurables such as empathy, clinical reasoning, and communications.

Think about a specific example: Berenson was a member of an Institute Medicine Committee that recently reported that physicians make wrong diagnoses 5 percent to 15 percent of the time. But, as he notes in a Jan 13 essay in The Journal of the American Medical Association (JAMA), we can’t measure the accuracy of diagnoses. So a critically important skill—the ability to correctly diagnose a disease– is excluded from performance-based pay even as Medicare carefully measures other processes than have no effect on health outcomes.

Paying docs and hospitals for the quality of what they do is a great idea. But until we do a better job measuring the quality of care that matters to patients, we ought to tread carefully—and perhaps look at other ways to separate good providers from the rest.

(Full disclosure: I am a unpaid board member of a community hospital and of The Armstrong Institute for Patient Safety and Quality).