Ranking NBA Refs By Their Willingness to Change Their Mind

Plus a Q&A with knife_guy -- a Bill Simmons memeographer

Mar 05, 2025

During All-Star media availability a few players were asked what NBA rule change they would make if they were the commissioner.

Jaylen Brown said he would allow players to financially invest in their own teams.

Anthony Edwards — who leads the league in technicals — said he would change the rules around technical fouls.

Kyrie Irving said he would like more days off (*monkey’s paw curls*).

But the suggestion that caught my attention the most was what Cade Cunningham proposed.

The star player for the Detroit Pistons said, “If you challenge a call, the ref that made the call, can’t be there to review it.” (h/t Jerry Donatien)

For those that don’t know, when a team decides to challenge a call, the lead referee (known as the Crew Chief) goes over to the monitor and looks at some replays before deciding whether to overturn the call or let it stand as is.

Cunningham suggested that Crew Chiefs shouldn’t be allowed to review their own calls. So if Scott Foster calls a foul on LeBron James and the Lakers challenge it, Cunningham thinks Foster shouldn’t be the one that gets to decide whether his original call was right or wrong.

Naturally, it would make sense that Crew Chiefs would be less willing to overturn their own calls compared to the calls made by other members of their crew. After all, who wants to admit they made a mistake? There’s a reason they don’t let us grade our own tests in school.

But when I looked at the data to see if Crew Chiefs are more (or less) likely to overturn their own calls I found something I wasn’t expecting.

To explain, I need to take readers inside the wonky NBA analyticsverse for a moment.

if you don’t care, you can just skip this next part.…..

The NBA publishes play-by-play data for every game. It describes everything that happens on the court. Like, which referee called a foul on the play, which is how I was able to put together a database of referee behavior.

The following screenshot is from the play-by-play of an early season game between the Bucks and Sixers. In the play-by-play we see that Caleb Martin was called for a shooting foul (S.FOUL) by the referee Marat Kogut (M. Kogut). The 76ers called a timeout to challenge Kogut’s call, but the challenge was unsuccessful and it resulted in Giannis shooting two free throws.

So I naively thought I could just look at the play-by-play to see which referee made the call that led to a challenge and then see whether the call was overturned or not. With that approach I thought could see which referees have their calls challenged the most and whether Crew Chiefs are more (or less) likely to overturn their own calls.

Honestly, I thought this was going to be a slam dunk idea. I expected to find some insight that either supported Cade Cunningham’s suggestion or dumped cold water on it.

But then I ran into a problem.

Anytime a challenge is successful (i.e., overturned), the NBA goes back and scrubs the play-by-play data in way that makes it seem like the original call never happened. That’s because the NBA updates the play-by-play based on the outcome of the challenge.

Let me show you an example from the same Bucks and Sixers game from before.

In the following play-by-play we see that Yabusele steals the ball and next thing you know the Bucks have challenged something on the court.

The play-by-play data doesn’t tell us what the Bucks challenged. To find out, I had to go back and watch the video of the game.

Here’s what happened:

After Kyle Lowry bricks a three, Brook Lopez grabs a rebound but it pops loose and it’s picked up by Yabusele (the steal we see in the play-by-play). Yabusele then dishes it to Andre Drummond who rises up for a dunk on Brook Lopez who gets called for a foul.

Neither the shot attempt by Drummond nor the called foul by Lopez appear anywhere in the play-by-play data — but that’s the action that led to the challenge. The called foul on Lopez is what the Bucks asked to be reviewed. But you can’t find any trace of it in the play-by-play. Let alone which referee called the foul on Lopez in the first place.

All you see is the result of the challenge, which is Drummond getting called for an offensive foul on Lopez. The called foul on Lopez is wiped from the record and all were left with is the result of the challenge.

and we’re back

So to summarize all that briefly: when a referee calls a foul that is subsequently overturned, the foul they originally called is erased from the play-by-play data and there’s no public record of the referee blowing their whistle.

There have been more than 1,000 challenges already this season. So unless you go back and watch the video clips of all 1,000 challenges to pinpoint which referee called the foul, you wont be able to see which referee has had their calls overturned the most — let alone whether Crew Chiefs are more (or less likely) to overturn their own fouls.

Which is to say I can’t find any kind of evidence to say whether Cunningham’s suggestion is a good one or not — at least not with what’s publicly available. I still think it’s a good idea even if I can’t prove it.

Okay, I buried the lede.

There’s still valuable referee-related insights to be found in the challenge data.

Even though we don’t know which referees have had their calls overturned the most, we do know which Crew Chiefs have overturned their own crews’ calls the most.

Let’s go back to our example of the foul on Lopez that was later changed to an offensive foul on Drummond. Even though Scott Foster wasn’t the referee that called the original foul on Lopez, he was the Crew Chief that responsible for deciding whether to overturn the call. So we can credit Fosters’s crew with a challenge and an overturned call. We lose some specificity when looking at things at the three-man crew level, but it’s the best we can do with the data we got.

Here’s a ranking of all the active Crew Chiefs, sorted by the percentage of challenges they faced that resulted in an overturned call during the last two regular seasons.

From this table, we can see Zach Zarba has overturned his crews’ the most. Of the 122 challenges Zarba’s crews have faced, he’s overturned 73.8 percent of them. That’s high considering that 60 percent is the league average.

Meanwhile, Brent Barnaky’s crews have overturned less than 40 percent of the challenges they’ve faced (extremely small sample size caveat).

But, this is looking at all challenges. And not all challenges are intitated equally.

Certain challenges have a much higher rates of being overturned than others.

Challenges that are triggered by an Out of Bounds or Goaltending Call are decided by the replay center, not the Crew Chief. That might partially explain why those calls overturned more 75 percent of the time. But also, there’s just less room for subjectivity when deciding who the ball touched the last. Fouls on the other hand are a different animal. Referees can always find a some form of contact to justify a called foul or they can whip out the trusty “marginal contact” language to decide something wasn’t actually a foul.

So to make sure we’re not picking up on Crew Chiefs that happened to rule on an abnormal number of Out of Bounds Calls that were decided by the replay center, we can limit our view to challenges of offensive and defensive fouls. That way we get a better picture of which Crew Chiefs have overturned the most subjective calls.

I think there’s a few ways you can look at this data.

The best Crew Chiefs don’t have their crews’ calls overturned. If we assume every ref is doing the best they can to be the most impartial they can, then it might be fair to say that Crew Chiefs with low overturn rates oversee the best run referee crews. In other words, their low overturn rates are proof that they get their calls right.
Crew Chiefs with high overturn rates are willing to change their mind in light of new evidence. Meanwhile, Crew Chiefs with low overturn rates are stubborn and/or want to stand up for their crew mates. This is the theory you’d be drawn to if you believe that some refs are bad actors.
The sample sizes here are too small to say anything meaningful.

On that last point, I tried to look at the data over a longer time horizon, but before 2022-231 it is not obvious which challenges were triggered by an offensive/defensive foul vs. an Out of Bounds call. So the following table is reporting the rates based on all challenges and not just ones triggered by an offensive or defensive foul. Which means it includes the challenges that were decided on by the replay center.

Looking at Scott Foster’s placement on this table I’m reminded of a line from the Big Lebowski that neatly summarizes my feelings on all of this.

Among the 30 active Crew Chiefs, Foster’s crews rank 30th in overturn rate over the last five seasons.

A re-refreshing Q&A with knife_guy, hall of fame r/billsimmons poster

One of the things I’ve enjoyed most about restarting this newsletter is the chance to talk to people in and around the NBA. I’ve done Q&As with Dean Oliver, Kevin Pelton, and most recently, Todd Whitehead. You can view the full archive of Q&s below.

If you scroll all the way to the bottom of that page, you’ll see a link to a Q&A I did in 2021 with someone only tangentially connected to basketball — knife_guy.

knife_guy makes memes about Bill Simmons on the r/billsimmons subreddit. I know that sounds lame, but if you’ve heard Bill’s voice more than anyone else’s outside of your extended family then maybe you’ll understand.

knife_guy hasn’t made memes in years and has all but stopped commenting on the Bill Simmons subreddit. But I’m re-running the original Q&A I did with them because the Bill Simmons subreddit has exploded in growth in the past few years.

We talked about Bill Simmons, r/billsimmons, and the Bill Simmons Podcast Podcast.

(Read to the end for a life update on our knife_guy)

This interview has been lightly edited for length and clarity

F5: I once saw a post on r/billsimmons where a redditor asked if someone could explain who knife_guy was. The top reply was, "He’s basically an autistic homebrewer and cigar fan that might murder Bill Simmons one day in a Mark David Chapman type of thing." On a scale of 1 to 20, how accurate is that description of you?

KG: The murder part seems pretty unlikely. I don’t think I’m on the spectrum, but could be? The rest is exact. So, 12? My mom diagnosed my uncle as on the spectrum (she is not a doctor), and I consider myself more neurotypical than he, so, ipso facto, 13. I take it as a compliment though. I placed 7th in a poker tournament once.

F5: I think a lot of the confusion about who you are is that your post history on Reddit is bizarre. Ten years ago you were posting questions on r/homebrewing and r/cigars. Then there’s this eight year gap of silence before you re-emerge and start posting, for lack of a better way to describe it, deepfried memes on r/billsimmons. What's that all about?

KG: I guess I didn’t have anything to say. I used Reddit as a resource much more than I used it as a way to interact.

Then I started to really track some of the inane things Bill would say on his podcasts, the crutch phrases, malapropisms, tortured analogies and the like. At some point I thought it would be funny to point them out by juxtaposing his words with the kind of saccharine motivational phrases prevalent on Instagram at the time.

F5: Bill does this thing when he's interviewing actors & actresses where he pulls up their IMDB page and reads their filmography to them out loud while his guest stares blankly back at him. I'm not going to do that. But I do want to ask you about your evolution as a memer. Your first post was a really low effort meme before you started moving into more abstract areas and eventually into full on mini-movies.

KG: The first one was really rudimentary, about the Kawhi trade. It wasn’t a quote, just a meme to comment on what he thought was emergency pod worthy. It wasn’t really well received, and I was bummed. I’ve been reading Bill since at least 2003 or so. I was one of the cliché guys who would print his Page 2 stuff and read it in the commode at work. But, it was clear his quality was slipping. And, I thought, are all the people in this subreddit—not really knowing how the subreddit worked—just followers who show up to worship Bill instead of thinking about his work critically? That kind of stinks.

Some of the early memes were really legit fried. I like that aesthetic. The problem is, a good hard fry can reduce the legibility or clarity of the point. And, so I started backing away from frying the stuff, so you could see the quote or, later, his face or, most recently, the lips of the person talking. On the other hand, if the image or images aren’t a little fucked up, it’s like it’s too good. These aren’t meant to read as pristine.

Actually, that’s one of the things about Bill that gets on my nerves. He seems so un-self-aware. Others on the subreddit have pointed this out much more intelligently and elegantly than I have, but he seems to have no ability to view himself from his outside. So, I wanted to degrade the imagery to depict him in ways contrary to how I imagine he perceives himself. He doesn’t know it, but he’s Johnny Lawrence, he’s Stan Gable, he’s Troy from the Goonies. And I don’t like that, because he started as the outsider you wanted to throw shit with.

F5: There was a story from 2020 on The Outline about the Bill Simmons subreddit where the author described discovering the forum as "like reading an astral projection of a conversation I’ve been having with myself for a decade. Finally, other people who have a vested, entirely unjustifiable connoisseurship in bespoke Simmons weirdness."

Did you have a similar experience or did you show up and think, this place sucks?

KG: I started doing this podcast with my pal about the Bill Simmons Podcast. It was (and kind of still is) only about the Bill Simmons Podcast. We named it the Bill Simmons Podcast Podcast. That led to some issues with our then podcast host (Anchor) thanks to some legally dubious claims by Spotify.

But my buddy and I used to just bullshit about Bill and his podcast and make fun of him or debate (not “litigate”) his theories. Then we decided to make our conversations into podcasts. We were trying to figure out how to connect to other smark Bill Simmons Podcast fans, and I think my pal pointed me to the subreddit. The first time I read a post on an episode, I was like, yeah, that’s how I feel.

F5: I don’t think people who haven’t listened to your podcast will fully appreciate just how niche it is based on the way you just described it. It's basically just you and your buddy riffing on Bill's mispronunciations and recapping his latest podcast episodes. I think there was a period where you guys were clipping together Bill's on-mic swallows, half-burps, and other weird mouth noises.

KG: Yeah, we don’t post much, but we still argue about his logical conclusions or insult his comparisons and then it’s like, “listen to this loud gulp into the mic.” Or, “listen to how sinister his laugh sounds slowed down.” I’m an idiot. I’m a child. It’s funny when he says “load” over and over.

F5: You said you had some issues with your podcast platform, Anchor, due to some legally dubious claims by Spotify. What’s the story there?

KG: That was really disappointing. We had been using Anchor from around April 2019 to October 2020. It's a pretty foolproof platform and its built-in ads made us a few hundred bucks over that time. And, it was free! Then Bill sold the Ringer to Spotify and Spotify also bought (or formed some sort of affiliation with) Anchor. Anchor then sent us an email that reads as follows:

Hi there,
We have received a complaint that the following content infringes the intellectual property rights of Matt Lieber of Spotify. While this claim is under investigation, this content has been taken down.
Show Title: The Bill Simmons Podcast Podcast Feed URL:

After a few requests, Anchor was pretty cool about paying us what we'd earned from the show, and they allowed us to migrate the feed to our current host, Transistor.FM, but the takedown was a complete capitulation.

F5: Did you ever try reaching out to Bill, The Ringer, or Matt Lieber to resolve the issue?

KG: No. It never occurred to me. It's a cool idea.

F5: Every now and then, when one of your memes doesn’t do well, someone in the comments will crack a "This is like watching Jordan on the Wizards" joke, but then you're back the next week with a Simmons / Indecent Proposal crossover (one of my personal favorites). When you're over there on your computer, doing knife_guy things, do you ever think to yourself, this is gonna be a top 6 or 7 post in the history of r/billsimmons?

KG: No. Never. Usually I'm just like, I guess I'm done. I do like any time Bill’s head is thrown back and his eyes closed. I also liked the “Rembert like compilation.” But I never know what people will think, and I'm always pretty anxious.

F5: Other people have recently started posting Simmons memes on the subreddit, but they’re not like yours. You can always tell something is your work without even looking at who submitted it based on the style. Simmons would probably call you a unicorn.

KG: I think unicorn implies greatness. I'm just a plugger. Also, not sure Bill would call me anything. I think he's loathe to admit the existence of criticism. "Stet all changes."

F5: Do you think Bill has seen any of your memes?

KG: I flatter myself by thinking maybe he has. Realistically, though, my guess is, at most, he’s scrolled by them without paying attention while keeping an eye on a soccer game.

F5: Yeah, I have a hard time believing Simmons browses the subreddit. I don’t think he would like what he would find. However, I’ve always felt that the subreddit is a fairly diverse place by Reddit’s standards.

KG: It’s really to Bill’s credit that the subreddit is diverse. He’s lost his fastball, but that muscle memory is there. He can still surprise us. I laugh out loud a couple times a week. I love seeing other memes or different approaches to criticism. Some people are really smart and thoughtful. The subreddit is cool in its self policing, too. You can disagree, but the assholes get rooted out. I recently started searching for this one poster's episode breakdowns, firewarner. He's doing a "thing" tracker and has his favorite quotes. Love it.

F5: If Bill came to you and asked what he could do to make his podcast better, what would you tell him?

KG: Hire an actual producer. Someone who can say, “try that take again” on botched ads (pronouncing COD as “cod” is a recent one), someone who can edit out or point out the crutches and weird words (I loved his brief flirtation with “centrifugal”), someone who can confront him with his sense of entitlement and self contradiction; mostly someone to rein in his negative (lazy, complacent) habits and get him to emphasize his outsider potshots. He should have been what Deadspin once was but as a podcast, the network TV version of Dr. Thompson, not someone who dreams of being accepted by the mainstream and eating lunch at the cool kids’ table.

Present Day

A few weeks ago, I reached back out to knife_guy to see what he’s been up to.

F5: Where have you been? Your last comment on the sub was 2 years ago.

KG: Same old stuff. Sitting at home in the dark churning podcast tape. Recording unreleased BS Pod Pod episodes on the weekly.

F5: I’m not sure how much you still follow the sub, but it’s really taken a turn for the worse recently. Does any part of you feel like the Oppenheimer meme?

KG: I have a long and storied history of denying responsibility. There was a quote a couple years ago, well after my posting rate had dropped to near zero, where someone posted that idea. I felt bad. But, it gives my silly bullshit far too much credit. I'm gobsmacked too see the borderline exponential growth in the sub. It's gone from like 10k to nearly 80k? I haven't notice a substantial downturn in quality, but I have seen an apparent higher incidence in mirror posts. I think that speaks to the high level of involvement and desire to be part of the conversation. But, probably also just to there being so many more subscribers (is that the right term?). Hopefully this psedo-intellectual response covers up the fact that it's gross speculation and conjecture.

Also, I'd like to substitute the word "churn" in my previous response with the word "grind" - I think it's more Russilo-y.

F5: Are you intentionally not podcasting/meming anymore because youve got nothing to say / because youre busy / stopped listening / or other?

KG: I'm still listening. I just ran out of time. That's not true. I lost the verve for the memes. This is true: I have a private Instagram account where I do similar memes based on people I work with. I've done about 1200 posts. It has two followers. But, for recording pods, it's the time piece.

F5: Bill was recently talking about how if he has a hall of fame eater in his house he wants to make sure he has a good chef. I heard that and was like, damn thats good knife_guy material. Have there any other moments on the pod, recent or otherwise, that you felt like would have been good material for you

KG: Yes, every episode. Anytime he starts talking about food, my ears perk up.

F5: Last time we talked I asked what you thought Bill could do to make the podcast better. You said:

"Hire an actual producer. Someone who can say, “try that take again” on botched ads (pronouncing COD as “cod” is a recent one), someone who can edit out or point out the crutches and weird words (I loved his brief flirtation with “centrifugal”), someone who can confront him with his sense of entitlement and self contradiction; mostly someone to rein in his negative (lazy, complacent) habits and get him to emphasize his outsider potshots."

How would you rate his performance since then.

KG: I truly think he’s improved. I think the infrastructure has helped. I’m interested to see if the video pivots results in a backslide. I’m very uninterested in that medium - I’m usually a car listener - but there must be a market for it.

I’m also pleased to see way fewer celebs, but his guest diversity and variety leaves something to be desired. Who was his last female guest, aside from his daughter? His cross promotion of ALL his company’s other shows is weak

To know if a challenge was triggered by an offensive and defensive foul I used this data compiled by the NBA, which I’ve only been able to locate for the last two regular seasons.

ivan Villanueva

Mar 6, 2025

Yo también publiqué algo parecido el 21 de febrero

https://x.com/elcheff/status/1893028547200954833

Chris Gunther

Really interesting and well done analysis! Regardless of the ultimate interpretation of the variation between crew chiefs, I’m surprised at how big it is. 48% to 66% is big!

The F5

Discussion about this post

Ready for more?