RIP RPI: The NCAA Evaluation Tool will Replace RPI as the Sorting Tool for Women's Basketball

[Ignatius L Hoops]

NCAA Evaluation Tool to replace RPI as team sorting tool for women's college basketball

The NCAA Division I Women’s Basketball Committee announced that beginning with the upcoming 2020-21 season, the NCAA Evaluation Tool will replace the Ratings Percentage Index as the contemporary sorting tool used to measure a team’s quality and help evaluate team resumes for selection and...

www.ncaa.com

No more RPI thread:

The NCAA Division I Women’s Basketball Committee announced that beginning with the upcoming 2020-21 season, the NCAA Evaluation Tool will replace the Ratings Percentage Index as the contemporary sorting tool used to measure a team’s quality and help evaluate team resumes for selection and seeding in the Division I Women’s Basketball Championship.

“It’s an exciting time for the game as we look to the future,” said Nina King, senior deputy athletics director and chief of staff at Duke, who will be chair of the Division I Women’s Basketball Committee in the 2020-21 academic year. “We felt after much analysis that the women’s basketball NET, which will be determined by who you played, where you played, how efficiently you played and the result of the game, is a more accurate tool and should be used by the committee going forward.”

The use of the women’s basketball NET was approved after a lengthy evaluation process, which included a comprehensive assessment of the strengths and weaknesses of the men’s basketball NET that has been used by the Division I Men’s Basketball Committee as a sorting tool since the 2018-19 season. After an analysis of women’s basketball statistical data over a 10-year period by a team from Google Cloud Professional Services, the Division I Women’s Basketball Committee concluded the NET algorithm built exclusively for women’s basketball was an optimal ranking tool and should be used beginning with the coming season.

Frequently asked questions about the DI women's basketball NET

Here's the information for all DI women's basketball NET questions, including what the NET measures.

www.ncaa.com

Some of the FAQ's from the NCAA release:

What is the NCAA Evaluation Tool (NET) for women’s basketball?

The NCAA Evaluation Tool (NET) for women’s basketball is the contemporary sorting tool used to measure a team’s quality and help evaluate team resumes for selection and seeding in the NCAA tournament. NET ranking is determined by who you played, where you played, how efficiently you played and the result of the game.

What components does the Women’s Basketball NET include?

The women’s basketball NET includes Adjusted Net Efficiency and Team Value Index.

What is meant by Adjusted Net Efficiency?

Adjusted Net Efficiency is a measure of a team’s overall performance during the regular season, determined by the difference between offensive efficiency (points per possession) and defensive efficiency (opponents points per possession). It also accounts for strength of opponents (as measured by their adjusted net efficiency) and location (home/away/neutral) of the games (against Division I opponents only).

What are the differences between the women’s basketball NET and RPI?

The women’s basketball NET is a contemporary sorting tool that more accurately measures the quality of a team determined by who you played, where you played, how efficiently you played and the result of the game.

The Ratings Percentage Index (RPI) was created in 1981 to provide supplemental data for the Division I Men’s Basketball Committee in its evaluation of teams for at-large selection and seeding of the championship bracket. The Division I Women's Basketball Committee began using the RPI in 1984. Simply stated, the RPI provided a ranking of each institution based on their Division I winning percentage and strength of schedule.

RPI consisted of three factors weighted as follows:

1. Division I winning percentage — 25 percent of the RPI
2. Opponents’ winning percentage — 50 percent of the RPI
3. Opponents’ opponent winning percentage — 25 percent of the RPI

Will RPI be used alongside NET going forward?

Beginning in the 2020-21 season, the RPI will no longer be used by the Division I Women’s Basketball Committee.

 

[Ignatius L Hoops]

A couple of notes:

“While the men’s and women’s basketball NET share high-level goals and individual components, the NET algorithm used in each is different,” said Lynn Holzman, NCAA vice president of women’s basketball. “The machine learning model developed for each sport utilizes only that sport’s data. The women’s model uses only women’s game data, while the men’s model only uses men’s game data. The weights for each are trained using the historical data from the respective sports with each accurately measuring the quality of a team.”

...

The women’s basketball NET rankings will be provided publicly on a daily basis on ncaa.com and ncaa.org starting in early December and continuing throughout the upcoming 2020-21 season

 

[CutDownTheNet]

Well, can't say that I'm gonna grieve the departure of RPI. It was pretty horrible.

What I find interesting (and a bit scary) is that, like the Men's NET, they seem to be equally mysterious and total lack of transparency about how the NET actually works. For the men, they gave a bunch of factors, but refused to say how they are combined. For the women (so far, unless we can find a good link on the NCAA site, which might be possible) they aren't listing even the complete list of factors (but it does look like net delta efficiency is a main factor, along with home/away/neutral).

The more interesting thing to me is that they admitted they are using machine learning. For the men too, but differently run models against their respective data. I'm not sure whether or not that was a well-published fact (about the men's NET using machine learning) or not. I studied the NET a little bit at the time it came out for the men, and I remember thinking that it's gotta be using machine learning, but I could never find anything that explicitly said that.

I also remember at the time being very concerned that the NET might be equally bad (but in the opposite direction) than RPI was. For one thing, the various factors had quite a bit of redundancy. Another issue was that it looked to me that it would be very easy for the NET algorithm to give too much credit for whipping some po-dunk college that has a horrible team.

In other words, it might be able to play the NET system in an opposite manner than one might play the RPI. Namely, whereas we've now (under Whalen) made good strides toward getting more competitive opponents such that our RPI is not dragged down by playing really bad teams in the non-conf season, under NET it might be advantageous to play really bad teams in the pre-season.

The one thing that ticked me off when the men's NET came out (and still ticks me off with the new women's NET) is the complete lack of transparency by the NCAA about how the whole NET thing works. As far as I know, it's impossible for one to compute their own NET scores, because they don't publish the algorithm.

I suspect that's intentional. If you don't publish the algorithm, then nobody can criticize the algorithm. It's kinda like, in analogy to the US voting system, at election time all the votes went into a complicated computer algorithm that nobody knew how it worked, and the system output the winner of the election, but never mind how the votes were counted, that's too complicated to explain to mere citizens, just take our word as to who won. That would trigger post-election protests. For the NET it's kinda the same thing. Here's the results, and no we are not going to tell you how they got calculated. It all seems rather convenient for the NCAA. Regardless of whether NET is any good or not (and as I noted, I have some strong suspicions that it might not be so good), the NCAA has at least eliminated the phenomenon of fan's complaining that it's being done horribly, since the fans are inherently incapable of constructing any rational criticism of the NET system, since you can't really criticize something in which they won't tell you the algorithm of how it works. Very convenient for the NCAA. Got rid of those griping fans.

I don't see why they don't publish the results of the machine learning model. Granted, not everybody is going to understand it. But people like me, that understand machine learning, could interpret how it works in layman's terms. So I don't know if they will eventually (or maybe they have already) publish the machine learning model for men's/women's basketball. But they should. In the name of transparency.

Otherwise, it's quite possible that the new NET evaluation tool(s) might be just as bad as RPI - with the only difference being that this time, us NCAA "customers" won't be able to prove how bad it is, since they refuse to divulge a copy of the machine learning algorithm such that we could evaluate it, and either bless it as "good" or point out whatever faults it might have. Just because machine learning is complicated, is no reason for the NCAA to completely withhold transparency.

 

[Ignatius L Hoops]

DI men’s basketball committee announces change to NET in 2020-21

The NCAA Division I Men’s Basketball Committee announced a change to the NCAA Evaluation Tool metric, which debuted during the 2018-19 season and replaced the RPI as the primary sorting tool.

www.ncaa.com

According to yesterday's Men's Basketball Committee release: The changes to the men's NET will make it "consistent with" the women's NET. (Emphasis added) Removing a basic metric like scoring margin seems like a bad idea; but that's just me.

No longer will the NET use winning percentage, adjusted winning percentage and scoring margin. The change was made after the committee consulted with Google Cloud Professional Services, which worked with the NCAA to develop the original NET.

“When we adopted the NET in 2018, we had reviewed several seasons worth of data and we insisted that we would continue to evaluate the metric,” said Dan Gavitt, the NCAA’s senior vice president of basketball. “We’ve been very satisfied with its performance thus far, but it became evident after two seasons of use that this change would be an improvement. While we will continue to monitor the metric, I don’t anticipate any additional adjustments for several years. We believe this change will result in more precision throughout the season and will be easier for our membership and the public to understand.

The updated NET is consistent with the women’s basketball NET, which was revealed last week after the Division I Women’s Basketball Committee worked with a team from Google Cloud to evaluate women’s basketball statistical data for a 10-year period .

In addition, the overall and non-conference strength of schedule has been modernized to reflect a truer measure for how hard it is to defeat opponents. The strength of schedule is based on rating every game on a team's schedule for how hard it would be for an NCAA tournament-caliber team to win. It considers opponent strength and site of each game, assigning each game a difficulty score. Aggregating these across all games results in an overall expected win percentage versus a team's schedule, which can be ranked to get a better measure of the strength of schedule.

 

[CutDownTheNet]

DI men’s basketball committee announces change to NET in 2020-21

The NCAA Division I Men’s Basketball Committee announced a change to the NCAA Evaluation Tool metric, which debuted during the 2018-19 season and replaced the RPI as the primary sorting tool.

www.ncaa.com

According to yesterday's Men's Basketball Committee release: The changes to the men's NET will make it "consistent with" the women's NET. (Emphasis added) Removing a basic metric like scoring margin seems like a bad idea; but that's just me.

It could be that the several metrics they threw away, per recommendation of the Google machine-learning consultants, were largely redundant metrics whose "information content" was also embodied within other metrics. That would have shown up as those metrics having very insignificant factors in the fitted machine-learning model; and forcing them into the model might have resulted in over-fitting to the historical data, which would not be good either. Also, it doesn't surprise me that they deemed a couple metrics superfluous, since when I looked at the men's metrics last year, I did see what I thought was some redundancy (which can be worse than using the best of the non-redundant metrics).

The margin of victory, if it's the one I think it is, was useless in my opinion anyway. If I recall, it was margin of victory capped at 10 points. But even in a close game going down to the wire, the winner often wins by at least 5 points due to intentional fouls, which might generate a 5-point margin of victory right there. So how much better is a 10-point victory than a 5-point victory? Well, something since it;s the difference between a near-tie and a 10-point victory. But is a 5-point-victory-but-near-tie any better than a 2-point-victory-but-near-tie? No. And is a 40-point victory better than a 10-point victory? Yes, a lot, but that metric wasn't capturing that since they capped it at 10 points. Yet such a 40-point victory was largely captured by other metrics. So what's the point of capping that one at 10 points when the other metrics that captured that were not capped. So I don't think that 10-point-capped margin of victory was contributing anything. And the analysts saw that this year in working with the women's data (which was 10 years, I think more than they originally worked with for the men's NET).

Overall, it sounds to me like maybe they made some improvements.

This part sounds interesting ... "In addition, the overall and non-conference strength of schedule has been modernized to reflect a truer measure for how hard it is to defeat opponents. The strength of schedule is based on rating every game on a team's schedule for how hard it would be for an NCAA tournament-caliber team to win. It considers opponent strength and site of each game, assigning each game a difficulty score. Aggregating these across all games results in an overall expected win percentage versus a team's schedule, which can be ranked to get a better measure of the strength of schedule."

I should reserve judgement until (and if) it is possible to review this in more detail, but without knowing any such details at this point, this at least sounds to me like an incredibly better (than RPI) method of accounting for the strength of an individual opponent.

One remaining concern for me (or maybe not so much a concern, since obviously they figured out some way to do this part, and perhaps more of a curiosity) is the fact that they were obviously using supervised machine learning, which in a nutshell means that they need some sort of expert-determined (and out-of-model) "quality-of-team metric for the machine learning to "learn." And where are they going to get that from. I've thought this through a few times in my head, and never got to a fully satisfying solution to the problem, but the partial solution is to look (for any given year) at the ex-post-facto tournament results of which teams beat which other teams. That at least gives a partial ordering on the 64 tournament teams. Maybe specific-league tournaments can help for those teams that don't make the big dance. So I can envision that it can be done, anyway.