• Share with:

Stars 101 explained the basic idea of Star Ratings.  This tutorial details of how the Star Ratings are assigned to each plan each year.  We start with the very basics -- weighted averages and build into sections around each technical area. The important part for most people is the list of “Implications” in the 3rd section.

A Weighted Average of Scores

The simplest definition of the Overall Star Rating is that it’s a weighted average of Plans’ ratings for individual measures. It’s very much like (probably inspired by) college grade point average. If you can remember that far back, you might recall two things. First, grades are assigned A, B, C, D or F for each class, and once the grade is assigned the underlying numeric value is set aside. It doesn’t matter if your “A“ was a 96% from one teacher or a 75% from a teacher who grades on a curve -- only the letter grade matters. Each class can have its own scale.  It would be a mess if you tried to average these scores, but the school pulls it off because it normalizes these systems into a common standard. The second thing to recall is that not all classes are equally weighted. It’s better to get an A in 3-credit Calculus and a B in 1-credit Tennis than vice versa.

CMS calculates Stars using the same type of weighted average. The first task is to collect scores for each of the measures from each of the plans and convert the numeric scores into 1, 2, 3, 4 or 5 Stars. The second task is to take the individual measure stars and calculate the weighted average and round to get a final Star Rating. I’m going to gloss over a whole bunch of details here so we can focus on the weighted average and rounding. Here’s a simple example with a few measures:

The weighted average is easy to calculate.  It's SUM ( Stars * Weight) / SUM( Weight)

The raw calculation for the overall rating is 34/11 = 3.091.  This rounds to 3 Stars.

Stars round on the half-star, not whole numbers, so the plan would need a raw score of 3.250 to move higher.  If C02 was 4 stars instead of 3, the rating would move to 3.18.  That would fall short.  But if that measure jumped to 5 Stars, or if a higher weighted measure added a star, it would move the average above 3.25 and put the Overall Rating up to 3.5.

If you’re new to Star Ratings, now might be a good time to pause and play with an example for a bit to get a more intuitive sense of it.  If you click on BHA’s Stars Planner, it will randomly pick a contract’s real data from the past year. See what happens to the overall rating when you move a 1x measure up and down a star.  Pick some higher weighted measure and get a feel for what it takes to move the overall rating.

Stuff Glossed Over

The calculation is not actually a 2 step process.  It’s more like 11 steps:

Assign Stars Per Measure

1. Collect and validate numeric scores from all plans for all measures
2. Calculate the numeric score Improvement Measure for Part C and for Part D
3. Use statistical methods to come up with cutpoints for each measure (the grading curve)
4. Convert each numeric score to a 1, 2, 3, 4, or 5 Star rating for each measure
5. Adjust certain CAHPS measure results up or down a star
6. Apply disaster area (and Covid) rules

Assign Overall Star Rating

7. Calculate the weighted average
8. Recalculate the weighted average without the Improvement Measure and apply the “hold harmless” rules.
9. Apply Categorical Adjustment Index (CAI) to adjust for plans with low-income members
10. Apply the Reward Factor that boosts plans with low variance and high mean ratings
11. Round on the half-star and award the final star ratings.

The first 79-page Technical Notes was published in 2012.  The 2022 edition grew to 175 pages as CMS has layered in multiple adjustments to the calculation.  Like any good rulemaking body, CMS listened to the critics who pointed out consequences of some part of the calculation and introduced adjustments to offset them.  We’ll use the rest of this chapter to discuss these adjustments.

If you’re not ready for all the details, stick around at least for the table in the next section.  We quickly summarize the “what” and “why” and try to answer the “what does it mean to you?”

Implications of Rules and Adjustments

Here’s a summary of the various adjustments and quirks in the calculation.  “Why” -- what is the theoretical reason for the rule, or what problem is it solving.  “What” - what is it’s impact on the calculation.  “Implications” - what does it mean practically to Plans.  Several of the quirks pertain to the 5x weighted “Part C Improvement” and “Part D Improvement” measures.  These are discussed in detail with the Improvement measure.

Setting Cutpoints

CMS explains how they use clustering to set cutpoints:

"Mean resampling is used to determine the cut points for all non-CAHPS measures. With mean resampling, measure-specific scores for the current year’s Star Ratings are randomly separated into 10 equal-sized groups. The hierarchal clustering algorithm is then applied 10 times, each time leaving one of the 10 groups out of the clustered data. The method results in 10 sets of measure-specific cut points. The mean for each 1 through 5 star level cut point is taken across the 10 sets for each measure to produce the final cut points used for assigning measure stars."

I couldn’t begin to explain this. Well maybe I could, but the Tech Specs themselves and a statistical textbook is the place to learn the theory and practice behind it.  Instead, here we’ll focus on a few of the impacts. 

1. Cutpoints are not Set Until After the Measurement Period. As designed, you cannot know the cutpoints until long after you can do anything to improve your score. This ensures that they’re fair in the sense that they represent actual relative performance across all Plans.  It also means that you can’t game things by aiming for exact targets and then stop.  Frustratingly, it also means that you cannot definitely predict where you stand.  Unless you’ve comfortably exceeded your target, you will have to operate using your best guess where Stars will land.  The actual rating will not be available until just before the annual enrollment period.
2. Cutpoints are based on “Clusters” not percentiles. This is the most surprising if you’re new to Stars.  The method gives 5 groups, and the Plans within each group have similar performance (as much as possible).  There have been years where only one Plan is 1-Star because no other Plan is close.  There have been years where cutpoint levels have been entirely skipped.  The count Plans at 1, 2, 3, 4 and 5 Star results are often very different across measures.
3. Some Measures are Harder than Others. As a consequence of the prior point, the average Star Rating varies from measure to measure.  CMS publishes the average each year in the “Fact Sheet” distributed with the technical specs.  In 2020, C28-Complaints About the Health Plan was the “easiest” with an average of 4.9 -- meaning that the vast majority of plans achieved 5 Stars.  C17-Falls Risk was the “hardest” with an average of only 2.5 Stars. 

Trends. The biggest mistake in predicting cutpoints is to assume that they move in one direction.  What you often see instead -- especially for new measures -- is that they will swing higher then swing lower.  If a particular cutpoint jumps 6% one year, it will probably give some of that back the following year.  A trustworthy prediction of the cutpoints will have to be able to predict both the direction of the industry and of the outliers.

You can get a feel for the cutpoint trends using BHA’s web tools.  Click on our Measure Browser and check the graphs at the bottom of the screen.

Measure Weights

Measures are weighted by CMS as follows:

There’s an important distinction between “process” and “outcome” measures.  Process measures are like Flu Shots and Eye Exams.  In these, the clinical standard of care is that a certain group of people are expected to receive a certain service.  Outcome measures are those that more directly reflect the members’ health, such as Blood Sugar Control and Hospital Readmission.  There is some grey area, for example Medication Adherence is considered an outcome measure.  Every year, CMS solicits public feedback and shares the responses, and the classification and weights of measures is a frequent topic.

Lately, the big buzz around measure weights has been CMS’s decision to increase the weight of the “patient experience” measures.  They had been 1.5X for years, but CMS announced plans to change it first to 2X and then to 4X.  The jump to 4X is a huge shift in the relative importance of different types of measures.  Every Stars focused conference has a presentation on what to do about it.  (Our blog does too.)

Reward Factor (r-factor)

CMS boosts the final rating for plans that have a “high average” and “low variance” in their individual measure ratings.  The high/medium is calculated from the 65^th and 85^th percentiles of Part C, Part D and Overall.  Variance is the measure of how consistent scores are -- a Plan with all 5-Star results is low (actually zero) variance, but so would be a plan with all 1-Star results.  A plan could average 4.33 from mostly 4-Star results and an occasional 5, or it could achieve the same average with nearly all 5’s but a few 1’s.  The first is low variance, the latter is not.

Fun Fact: Before 2015, CMS labeled this “Applying the Integration Factor (Reward for Consistently High Performance) and referred to it as the “i-factor.” In 2016 this truncated to “Applying the Reward Factor”.  Maybe not so fun, but it was news to me.  My old colleagues called it the "i-factor" and I never undestood why.

The Tech Notes define precisely want constitutes high/medium/low average score and high/medium/low variance.  Then the r-factor is as follows:

The Tech Notes provide a table of thresholds for “High” and “Relatively high” separately for Part C, Part D and Overall.  They vary, but it’s about 4.4 for “High” (85^th percentile) and about 4.1 for “relatively High” (65^th). The reward factor is jet fuel for getting to 4.5 and 5.0 Stars.

You can calculate backwards to figure out exactly how much scores can deviate from the mean.  But the important thing to note is that high variance gets no bonus.  CMS is rewarding plans for avoiding any poor scores.

Categorical Adjustment Index

I used to work for a Special Needs Plan(SNP).  Every year, we’d complain that our patients were sicker and that Stars were stacked against us.  It was true -- some Star measures were especially difficult for this population.  But CMS’s policy response was not to adjust for this risk -- sickest patients would be those that need the highest quality care.  Finally, in 2017 CMS threw us a bone with the “interim analytical adjustment called the Categorical Adjustment Index.”

Fun Fact:  CAI was introduced as an interim adjustment in 2017 “while measure stewards undertake a comprehensive review of their measures in the Star Ratings program”.  The word “interim” was quietly dropped in 2021, and in 2022 the review language was also dropped.  So, it seems CAI is here to stay.

For the CAI, the “sickest” patients are defined by two percentages:

• % of membership with “Disabled Status”
• % of membership with Low Income Subsidy or Dual Eligible status

CMS deducts 0.035526 points from the final score (approximately -3 weighted stars) for the least disabled/low income plans or adds as much as 0.156984 (+12) for the plans with the highest percentage of each. 

There are two takeaways to note:

• If you have a healthy and wealthy membership, make some allowance for CAI to tug at your Star Rating
• Star Ratings are hard for low income and disabled populations. CAI is helpful.  Don’t panic if the raw score is coming up short. 

SNP-Only Measures

Care for Older Adults measures calculate the % of membership who receive 3 services at least once during the year: pain screens, functional status assessments and medication reviews.  The SNP Care Management measure is the % of members who received their health risk assessment on time.  These measures only apply to Special Needs Plans (SNPs).

SNPs are by definition high on disability and low income membership.  The COA measures insist on following a basic standard of care for elderly and frail membership.  HRAs are always part of SNP plans’ responsibilities.  So, in addition to CAI, these measures exist to boost SNP plans’ Star Ratings.

The COAs are very simple to scan HEDIS charts for.  Whatever the PCP hasn’t completed can be completed in a single phone call from the health plan.  They should be a gimme.  

Merged Contracts

Many of the rules discussed originated as CMS's response to some quirk or loophole in Star Ratings.  The handling of Star Ratings in cases of contract mergers is an example.  What happens when a Plan (hcontract) ceased to exist?  It used to be that the rating ceased to exist too -- even if it’s a low score.  What happened if the company had another Plan that the members could be moved to, and that plan had a higher Star Rating?  Companies, of course, did so.  This practice was widespread, and for example, a 3 Star 200,000 member plan closed and the members were shifted onto a 2000 member 4 Star plan. 

CMS responded in 2017 with rules for “Mergers, Novations and Consolidations”.  This gave the merged contracts’ Star Rating as the average rating of all the prior contracts weighted by the membership.

Disaster Areas

Star Ratings are based on relative performance -- Plans compete with all other Plans in the country.  But with a local disaster (hurricanes, floods, wildfires, etc.) it’s an unfair comparison.  To adjust, CMS gives plans the prior year’s score instead.  The current year scores aren’t entirely discarded.  If the plan succeeded in raising the sore for a measure, CMS uses the better of the two scores. 

Affected contracts are defined as plans in which at least 25% of the Plans’ members are in a county in which there is a federally declared disaster “individual assistance area”.

There are a few exceptions; these are explained in the Tech Notes and not rehashed here.

COVID-19

The pandemic made the entire country effectively a disaster area and Plans eligible for relief.  The above rules apply to everyone in 2022, and CMS had to make some adjustments in how they calculate the averages for the reward factor and cutpoints.

Improvement Measure

CMS has two meta-measures included in Star Ratings:

• 5x - Part C Improvement
• 5x - Part D Improvement

The next chapter of this tutorial will be devoted to these two, but I want to point out some highlights before closing out this chapter. 

• A good Improvement Rating can push you above 4 stars. But a bad one cannot drag you below it.
• Improvement measure is calculated by the net of how many measures go up versus how many go down. It's designed to center at 3 Stars if an equal number go up as go down.  The up/down is based on the scores not the measure star so if cutpoints jump higher a measure could go down in stars but still be "improved".  Final note, the add/subtract is weighted by measure.  
• The change must be “statistically significant”. There is an interval around the prior year score of “unchanged”.  Below that is “decline” and above is “improved”.  You can calculate this in advance of HEDIS and know exactly what you need to hit -- changing cutpoints do not affect this calculation.

The last point is important.  Attention to Statistically Significant Improvement during HEDIS should be a critical piece of your Stars Strategy.

Next Up

We’ve now provided an overview of the whole Star Ratings program and a detailed explanation of how they’re calculated.  To complete the introduction, we’ll have a chapter on each of the types of measures (including Improvement) and a high level discussion of the types of things that Plans do to improve scores.