This week, The New York Times published an investigative report by Katie Thomas on Medicare’s five-star rating system for nursing homes.  Among its findings: Medicare’s Nursing Home Compare tool relies largely on self-reported data by the facilities themselves and is thus unreliable.

On one hand, this is a bit odd, since Medicare’s website explicitly describes these very shortcomings.  On the other hand, the article (which will get a lot of attention because it is The New York Times) may encourage consumers to look beyond Medicare’s rating system. And that would be a good thing.

But the real problem is not that Medicare takes facilities’ word for it when it comes to quality. The real problem is that too often it is measuring the wrong things. The lesson is simple: The five-star rating system is useful (if flawed), but a wise consumer should see it as only one tool in the search for the best possible facility. After all, who’d buy a car based only on its government fuel-efficiency rating?

Use Nursing Home Compare, by all means. But don’t stop there. Visit facilities and look beyond the wood-paneled lobby. Talk to residents and their families. Talk to nurses and, especially, to aides, who provide nearly all of hands-on-care.  Check the local ombudsman office for complaints. To be fair, the Medicare.gov website makes many of these same suggestions, but they are buried in a 56-page guidebook, so few will ever read them.

And remember too that facilities may be good at some things but not others.  One may provide great rehab for people with hip fractures. Another may have a poor rehab unit but do a terrific job with long-stay dementia patients.

It is important to remember that Medicare is mostly rating safety, not quality. Safety is an element of quality but hardly the entire story. And the easily-quantified measures Medicare does use say very little about whether a facility provides high-quality, person-centered care that responds to the individual needs of its patients or residents.

Here are three examples of key components of the rating system. They are somewhat oversimplified, but may help explain how the rating system can tell less than the full story:

Falls. You might say a facility that reports fewer falls is better than one that reports more. And often it is. But I can imagine a nursing facility that reduces falls to zero by simply not letting people walk around freely (perhaps through the use of chemical restraints). Your dad might prefer a place that gives him more freedom, even if that means increasing his risk of falling.

Weight loss. At first glance, this sounds logical. You wouldn’t want your loved one being starved. But weight loss, especially at the end of life, is a complex issue. Often, as people begin to die, they stop eating. And they inevitably lose weight. It is a perfectly normal response as the body shuts down.

Imagine two facilities with large populations at the end of life. One routinely encourages the use of a high-carb diet or even feeding tubes to keep patients’ weight up. Another encourages patients and their families to make informed choices about eating at the end of life. And in that facility, many more residents reject feeding tubes, lose weight, and die in relative comfort. Which is the “better” facility?

Staffing.  Medicare ratings measure only patient/staff ratios. Once again, it might seem that a place with more staff per resident is better than one with fewer. But Medicare does not ask about staff turnover, training, or morale. Do staffers care about their residents? Ratios tell only part of the story.

These flaws go far beyond consumer ratings. As pay for performance becomes more important, they will also increasingly be used to determine how much facilities get paid. To be blunt, if Medicare creates incentives (whether through ratings or direct payment) for a facility to hire lots of poorly trained aides instead of fewer high-quality staff, that’s exactly what it will do.

How would I use the rating system? Probably as an initial screen to help choose which facilities to look more closely. I’d eliminate facilities with consistent ratings of three stars or less. But I’d certainly not mindlessly choose one with a four or five star rating. As The Times correctly reports, it is too easy for facilities to game these numbers.  But mostly, it is because Medicare is measuring the wrong things.