ORCS Pairing Procedure Addendum Discussion

This addendum got swept under the radar even though it explains what Bernstein's motion entails. Hoping we can have some quality discussion regarding this:

http://www.collegemocktrial.org/Addendum%20A%20-%20ORCS%20Pairing%20Procedure%20(2019%20midyear%20motion%20EC02).pdf

I'm still highly skeptical as to why we should change to this type of procedure to reduce CS standard deviations when nobody has really had a major complaint regarding it. Yes, sometimes you can get tough schedules at ORCS, but you will always get a tough schedule at nationals anyways.

I agree that teams having unfairly tough schedules at ORCs isn't a problem that'll ever be resolved. Even with this new system- a team could be paired with 3 other teams that happen to better competitors than their TPR can predict- and lose a bid to Nats. But, we gotta keep in mind that this system also attempts to fix the opposite problem- an unfairly easy schedule at ORCs. Everyone bitched and moaned on MTC last year about Texas A&M being a "BINGO team" (which was gross, A&M is an incredibly sweet team who didn't deserve to be cyberbullied) and "evening out" CS could end those kinds of complaints.

Honestly, I'm kind of skeptical that this new system will do much to actually even out schedules. But I don't see any huge downsides. Especially as a former tab director, after reading over the procedure it doesn't seem like it'll be all that more difficult than the standard procedure. The only thing that gives me pause are the limitations placed on tab to resolve impermissibles for round 3 pairings. But if it might do something to make ORCs more fair, why not, yknow?

My concern about it is that the difference between the top seed and bottom seed might be drastic. For example, in the projected Memphis ORCS, 1. is Rhodes A and 6. is Alabama A. A team in another grouping is now equally likely to face Rhodes A and Alabama A and that's supposed to be equal weight. Similarly, the bottom groups could be drastically different as well. For group C, top ranked (13th) could be a very competitive team. Right now Cincinnati projections have Michigan State A in the same category as Michigan State B and playing those two teams could result in very different results. I think overall this could be effective, but it also could result in some crazy high/low CS's because of the teams you face.

To give you a more clear example, lets say a team at the top of the D section faces the number 6 ranked team and somehow beats them. You just beat the number 6 team and now you're guaranteed to not face a team higher in TPR for the rest of the tournament. You could end up facing the bottom team from B, C, and D and make it to Nationals with a decently low CS and decently easy schedule.

TPR is also fairly inaccurate in regards to various "dark horse"/rising teams (e.g. Juniata College A), where their TPR and seed likely won't reflect their actual skill level. This would likely lead to more complaints about how teams got "the hardest D team" or otherwise still had an insanely tough schedule.

I very strongly dislike this new process. I think that it is not going to solve the problem. I think it's extremely likely that some teams will have much easier schedules than others. I think it is incredibly unfair to have the teams in the bottom groups hitting the teams in the upper groups the first two rounds. If you are a "bad" team you have to face your two toughest opponents right away, while "good" teams get to start with two easier matches. I also am not a fan of the high-high pairings round 4. Wasn't that specifically eliminated to avoid a situation where two 4-2 teams don't knock each other out while a 3-3 team sneaks in because they get to hit an easier team round 4? I think this system causes way more problems than it solves.

While TPR is not a perfect measure of strength, something like this is necessary. For example, at the Chestnut Hill ORCS this year, Brown A hit Princeton A, Wesleyan A, Tufts A, and Harvard A ending with 4.5 ballots and a CS of 22.5. BC A, meanwhile hit UConn A, UConn B, UMass-Lowell, and BU B. BU B was the only bid-earning team they played to whom they dropped both ballots. BC then ended with the lowest record in their division at nationals. The new pairing system would hopefully prevent a schedule like Brown's from happening.

Like @angryattorneytm , I think I’m actually a fan of this process. I think being less random when it comes to strength of schedule at ORCS is better than being more random. It reduces the amount of luck involved in winning a bid. Is this method perfect? I don’t think so. Is it better than what we have in place at the moment? I think so, if our goal is to level out strength of schedule. I believe eliminating the random round 1 pairing and adding another metric of measuring strength gets us a little closer to that. In this new method, we organize schedules not just by performance, but also by TPR. Is TPR perfect? No, but it’s arguably closer to a true representation of teams’ relative strength than anything else we have. Having that extra layer creates and interesting dynamic; the tier-matching and power-matching balance each other out, mitigating some of the outlier effects relying on each one alone would cause.

This new system helps to avoid bad teams getting lucky by hitting only teams that are in their same tier, but performing similarly all tournament long, or getting unlucky by being a team that’s able to punch above their weight (let’s say they can snag a split from the best teams), but consistently gets paired with a top team that just slipped up the round before or got paired with another top team just before. We’ve all seen those 3-5 teams with an unfairly high CS. I think this system helps control for all those factors more than random pairing for round 1 with power-matching rounds 2 and 3.

My biggest question has to do with the high-high pairing for round across the upper tiers and lower tiers. Would like to know if Justin Bernstein has a deeper explanation other than to balance out CS. During the invitational tournaments, you usually see a bit more balanced CS spreads at full size tournaments than at regionals because they can pure power match round 4. At AMTA tournaments, for the reasons stated by people above, the reverse power matching with the “in” bracket and the “out” bracket helps mitigate mediocre teams sneaking in past better teams in round 4, especially when more than just the top prize trophy is at stake by way of multiple bids.

So yeah, why the high-high pairing in round 4 across A/B and C/D tiers? That’s a question I want a fully-detailed answer to before I could probably communicate a strong opinion on this. But as of right now, I think I do in fact like it.

I think this proposal is a really good one. Under the old ORCS system, it's way too easy for a team to sneak through without playing any hard teams and the another team to get completely screwed. The random round one pairings result in all sorts of weird things where bad teams wreck even worse teams and meanwhile good teams split with each other. That makes round 2 essentially random as well. It’s only in round 3 that you start to see some sorting out, but even then, you often get decidedly mid-level teams at 4-0 or 3-1 allowing a good team to slip by three rounds without ever being seriously challenged. And after round three there is power protection, so far less sorting can occur. This has resulted in weird things like teams making it through tournaments with an 8-0 record and a CS of like 8, which is ridiculous. It’s not the fault of those teams (you can only play who’s put in front of you), but it’s also not fair.

In order to illustrate the difference, I’m going to pick on two teams from last year: Yale A and Wheaton A (who are both great teams which is why I’m picking on them).

Yale A may have gone on to prove themselves later in the season, but their route through ORCS was insanely easy. They played no other teams that qualified. They started out with a random pairing. They played Colby (which was funny because of what happened at regionals but ultimately doesn’t seem to have been much of a challenge). They wreck Colby by +14, +14. Then they go on to play UMass (their highest ranked round by far). This is a 2-0 round, but partially because UMass has just wrecked Quinnipiac, who would eventually be an 8-0 team. Yale splits with UMass. Now they are 3-1, so you would expect them to get a hard round 3. Instead they play Princeton B. Princeton B has a relatively easy schedule so far (based in the final records of their opponents), and so is ranked high, but they will eventually end up with 3-5 (not exactly a huge challenge for the 5 time division champions). Yale wins again. Now, in invite or nationals pairing, Yale probably would have had to play another qualifier. But, instead, they get power protected and end up playing Williams, who ended with a 2-6 record. I’m not saying Yale was bad at that tournament, but their schedule wasn’t exactly the battleground we usually think of when we think of ORCS. Another way to think of it is that based on last year’s field, Yale had to play two C bracket teams and two D bracket teams. Sure, they went 7-1, but I’m guessing there are teams that did not qualify that could have qualified if they had that schedule.

Speaking of which, let’s compare this to what happened to Wheaton. Wheaton gets an unlucky first pairing against Ohio State A (who would eventually place 5th in their NCT division). They split. This, in and of itself, might suggest they deserve to be at Nationals, but that’s not how this works. They then go on to play NIU who are 2-0 (because side imbalance can do that). NIU has been to nationals multiple times in the last few years. Wheaton wins and wins convincingly (+15, +9). The world is not ready to give Wheaton a break. Unlike Yale, their 3-1 round pits them against another team who would eventually qualify for nationals: Cincinnati. Again, they split. Now they are 4-2 which puts them in the middle of the power protection bracket, and that runs them right into another team that has has some recent historical success: the University of Washington. They split again. They end up 5-3 having split with two teams that eventually qualified. By the current brackets, they played two A bracket teams and two B bracket teams. Relative to Yale, they just got really unlucky.

The new system avoids this kind of disparity. I know TPR isn’t perfect, but, in many ways, its more predictive than the current system of total randomness for the first couple rounds.

One thing I think they might do to make this system even better in coming years is find a way to factor regional performance into the ranking system so that they can predict current year changes in performance (someone graduated so the program got way worse, they got a new coach and are doing way better etc.).

bdopl wrote: My biggest question has to do with the high-high pairing for round across the upper tiers and lower tiers. Would like to know if Justin Bernstein has a deeper explanation other than to balance out CS. During the invitational tournaments, you usually see a bit more balanced CS spreads at full size tournaments than at regionals because they can pure power match round 4. At AMTA tournaments, for the reasons stated by people above, the reverse power matching with the “in” bracket and the “out” bracket helps mitigate mediocre teams sneaking in past better teams in round 4, especially when more than just the top prize trophy is at stake by way of multiple bids.

So yeah, why the high-high pairing in round 4 across A/B and C/D tiers? That’s a question I want a fully-detailed answer to before I could probably communicate a strong opinion on this. But as of right now, I think I do in fact like it.

Obviously, I don’t know what the board’s actual rationale is, but here is my guess. Suppose we let there be a high-low round. It’s the last round so the high low is the A/B C/D round. If you are doing really well in the A bracket, you will have to play a really hard A team in round 3, and a (relatively) easy B team in round 4. If you are a B team that’s doing really well, you will have to play a hard B team in round 3, and a (relatively) easy A team. In other words, it would mean there is a fundamental difference between how we treat teams based on what bracket they are in (rather than the current system where each bracket is treated pretty much the same).

Why does that matter? Let’s say that by actual strength (not predicted strength by TPR) you are the fourth best team in the tournament. You can beat anybody except the the top three there, all of whom are in A bracket (as they will be in most years). If that is the case, and the pairings are high-high for all rounds, then you don’t care which bracket you are in. You will end up probably having to play one of those three scary teams in your high-high round against A bracket. And then you will play some team from B bracket that is doing well relative to B bracket in your high-high round against B bracket. Your expected record is 6-2 (winning your D, C, and B rounds and loosing your A round).

But now suppose we do high-low in round 4. Now it matters which bracket you are in. If you are in A bracket, then you will have a high-high round against the A bracket in round 3, and probably have to hit one of those three scary teams. Then you get a very easy round against B bracket, but you were going to win any round against B bracket anyway. Your expected outcome is 6-2. But if you happened to be lower ranked coming in (maybe your team has a bad year last year), and got placed in B bracket, now you just got lucky. You get high-high pairing in your B bracket round and you win because you can beat anybody in B bracket. But now your A bracket round is high-low. So you miss those three scary teams and play one of the A bracket teams you can beat. Now your expected outcome is 8-0. In other words, if we high-low, it starts to matter a lot more which bracket you are in.

So if you are the fourth best team and the three teams that are better than you are in A bracket, you want to be in B bracket. I’m not going to go into full detail, but there other scenarios, where you would have the advantage if you were in A bracket (e.g. If you can’t beat anyone from A bracket but you can beat half the teams in B bracket, then you want to be in A bracket for the high-low because it makes your expected record 6-2 rather than 4-4). Some of these differences make the difference between a qualifying record and a non-qualifying record.

AMTA doesn’t want to make it the case that there is a fundamental advantage to being in one bracket or another inherent in the pairing system. This would incentivize all sorts of bad behavior (e.g. swapping your A and B team rosters to get the bracket you want) and would just be unfair.

I think it's an excellent system that makes things much more fair.  I might prefer to have it so that in a 2-2 tournament each team will play one top half team and one bottom half team each day.

Example

R1 A v. D, B v. C
R2 A v. A, B v. B, C v. C, D v. D
R3 A v. C, B v. D
R4 A v. B, C v. D

I like having A v. B in R4.  I think it makes a lot of sense to have the teams projected to be on the edge of earning a bid having to beat the teams projected to earn a bid to get through in R4.

One thing I am curious about is the ramifications of eliminating power protection in R4.  On first glance power protection sounds like its to protect the teams at the top.  But really what its for is to protect the integrity of the rounds that will determine bids in Round 4.  Without power protection you might end up in a situation in R4 where in one round a 6-0 A group team is playing a 4-2 B group team, the 6-0 team already has a bid locked up and they just fuck around and give the 4-2 team an easy sweep (or, maybe with a bid secured, they lay down in a round for the 4-2 who they either are friends with or think will be easier to beat at the NCT relative to the other 4-2 teams).  Meanwhile in another courtroom you have a 4-2 A group team playing a 4-2 B group team in an elite round for a bid, they split, they're both objectively better than the 4-2 team in the other round, but they get left out.  You've also got the same situations, perhaps more likely situations, in the R4 C v. D matchups.  Say you've got a 3.5-2.5 C group team playing for a bid against a dejected 0-6 D group team.  There's a real risk a 0-6 D group team, after having a hard weekend, won't give a real effort, or will decide to play the word game, or will decide to run that crap theory they've been joking about all year.  The same 0-6 D group team, sitting at 0-0, could have fought hard in round 1 against the A group team they played.  Now they're laying down to the bid-possible C group team, while in other courtrooms the A and B group teams are duking it out for bids that may not actually be in play because the C group team is guaranteed to win against a D group team that isn't interested in putting up a fight.  This is a tough situation, but I will concede I think it will be a very rare occurrence and its worth the risk to get what is a much more fair way to set ORCS schedules.  Maybe somebody better at this kind of stuff can figure out a way to still separate out the teams no longer fighting for a bid in R4 while keeping the overall structure of this system in tact.  

The last thing I would note, is that if we're seeding teams based on team strength, the next step should be to see if we can come up with a more accurate way of assessing team strength beyond just TPR.  For example, while using 3 years worth of data might be a pretty fair way of judging program strength on a more historical scale, and combatting the results of an outlier season, do we really think results from 2016 should be factored into seeding ORCS in 2019?  For most teams, there will be at max 1-2 (and in the overwhelming amount of cases, ZERO) people still on the A/B team that achieved the 2016 ORCS result competing on the same A/B team in 2019.  Should we not find a way to factor in performance at regionals, which are the freshest results and involve competitors that are definitely active this year?  Making a more accurate tool that is designed with seeding ORCS in mind, instead of using TPR which was designed to be evaluative of team strength at the end of a season andnot in the middle of a season, is the next step.

Now that the new pairing system is official, I went back through the 2019 ORCS and retroactively applied the tiers to get a better idea of what we can expect in March. The data is available here: https://docs.google.com/spreadsheets/d/1bIuqigAv4pbvMYNeJE1Q109MV1Y1XMM45AOkO3P85GI/edit?usp=sharing


Tier A
Overall Win Rate: 66.1%
TPR Rank Min: 1
TPR Rank Median: 27.5
TPR Rank Max: 64
NCT Qualifying Rate: 50.0%

Tier B
Overall Win Rate: 58.8%
TPR Rank Min: 40
TPR Rank Median: 84.5
TPR Rank Max: 162
NCT Qualifying Rate: 31.5%

Tier C
Overall Win Rate: 43.5%
TPR Rank Min: 97
TPR Rank Median: 167.5
TPR Rank Max: 241
NCT Qualifying Rate: 5.5%

Tier D
Overall Win Rate: 31.6%
TPR Rank Min: 191
TPR Rank Median: Unranked
TPR Rank Max: Unranked
NCT Qualifying Rate: 3.7%


Round 1
Under the new system, the tournament starts with Tier A v. Tier D and Tier B v. Tier C. 

As anyone would expect, A v. D turned out to be the most skewed match-up last season, with the average ballot going to the A team by +9.1. We can expect a lot of 2-0 A and 0-2 D teams after the first round, because Tier A won 88.6% of the ballots between them in 2019. The overwhelming majority of the ballots taken by D teams in this match-up came from drastic splits, where Tier A picked up one ballot by double digits and dropped the other by less than 5. It will be important for top teams to be able to start off with a clean sweep, but huge blowouts may be counterproductive. With most of Tier A expected to head into Round 2 with 2 wins under their belt (and 0 CS), high-high pairings against the more variable Tier C will depend almost entirely on PD.

The other section of Round 1, B v. C, should result in more splits and upsets. Last year, Tier B picked up 61.2% of the ballots in this match-up, with the average ballot going +1.1 in their direction. While this turns out to be the most skewed match-up between any directly adjacent tiers, the difference isn't by much and we should still expect varied results. A sweep by C teams here would be extremely helpful in reaching the 6 win benchmark, where losing a round 2 match-up to Tier A suddenly becomes affordable with only low-tier rounds waiting on the second day.

Round 2
The second round holds both of the only match-ups between teams that are 2 tiers apart: Tier A v. Tier C and Tier B v. Tier D

Results from this round should be similar across sections. Last year A won 76.9% of the ballots against C, with the average being +6.8 to A. B took a similar 81.6% of ballots versus D, with an average of +6.9. Teams in Tiers A & B are hard-pressed to end the first day with a perfect record through their easier competition, and anything less than 3 wins will make a Nationals push very difficult. Teams in C & D will be poised to decide their own fate if they can manage to go .500 combined against the top tiers, but that leaves no room for slip-ups in their bottom rounds. 

Round 3
Day 2 starts with the best on-paper match-ups, with teams competing against their own tier. 

We can expect a lot of close ballots, a lot of splits, and a lot of teams being knocked out of contention for Nationals. Since half of each tier will be put on Prosecution and half on Defense, Round 4 match-up possibilities will be narrowed to 3 total teams, and possibly even down to 1 or 2 because of schools who send multiple teams.

While these high-high pairing are determined by wins/CS/etc within the tier, it's worth noting that the higher ranked team won 57% of the ballots last year in same-tier match-ups. This bumps up to 62.6% if you remove group D, which is made up predominately of teams that were unranked on the TPR headed into ORCS. Surprisingly, TPR was actually more predictive of ballot outcomes within a tier, then between adjacent tiers (for tiers A through C). While it seems like the most even results should come in intra-tier rounds, when TPR rank should be closest, last year that wasn't the case. For example, in B v. B rounds, the higher ranked team won 66% of the ballots, but when B hit C, the (higher ranked) B teams only won 61% of the ballots. This was true in multiple cases, but never by more than 5%, and could be merely a coincidence in a relatively small sample size.

Another oddity was that TPR followed by regional performance (the tiebreaker used for tier assignment) was actually inversely predictive in D v. D match-ups, where regional performance would have to be used often to differentiate from the many unranked teams. The higher ranked team won only 43.7% of the time, which was the only type of match-up where ranking did not predict the overall skew accurately. While the theoretical implication is interesting for arguments that regional performance should be more important in deciding tiers, it doesn't make any actual difference under the system adopted as-is. D v. D will be paired based only on day 1 results, and last year every single unranked team would have been in Tier D. The tiebreaker of regional performance is very unlikely to be used for unranked teams, and in fact the tiebreaker wouldn't have been applied last year at all.

Round 4 
The final round is A v. B and C v. D


We can expect the most intense round to also be the closest. Last year, A won 54.7% of their ballots against B, and C won 60.4% versus D. On top of the existing balance, add in the fact that all these match-ups will be made even more competitive by high-high pairing after 3 rounds of competition, whereas last season they happened all throughout the tournament (including random R1 pairings and mixed R4).



Analysis
Overall, I think this change is a move in the right direction. The new system should drastically reduce teams being kept out of nationals by an extremely hard schedule. In 2019, 3 teams did not move on after managing a .500 record (or better) with a CS of 21.5 (or higher). Those teams all had match-ups exclusively from Tiers A & B. I think fairness is improved by including a tier system to the match-ups, and I don't see a better way to structure it. While it may still be possible to have an extreme CS in this system, only 5 nationals teams came from Tiers C or D, and never more than one at the same tournament. This means that teams are basically guaranteed at least one round against a team that isn't nationals caliber. Putting the closer rounds on day 2 gives the best chance for pairing to fairly sort the match-ups. 

Since every team shares a schedule structure with their tier, and the more lopsided match-ups come earlier, we can expect CS and PD to remain more relevant in pairing even on day 2. This should help balance out the variance between hitting the top/bottom of a tier. The system should also help ensure that when extreme CS does occur, it tends to be top teams who are strong throughout the tournament and manage to move on (in the high case), and teams who were eliminated before day 2 (in the low case). I think high-high pairing in round 4 is essential to keep a specific tier from being at an advantage.

As far as the tier make-up, I think AMTA was right to stick to TPR over Regional performance at this point. I would be concerned with creating incentives to throw rounds after clinching a spot if Regional records directly influenced ORCS match-ups. A more predictive way to rank might be reached eventually, but I don't think the current year's Regional placement is it.

My main concern with this system is that the nationals cutoff will go up (or fail to drop despite returning to 6 bids per tournament). Because TPR is predictive, last year there ended up being more rounds between the top tiers and more rounds between the lower tiers, as teams were paired with others who had similar results. Now that we'll mandate that each tier hit the rest exactly 25% of the time, I think a real possibility is a larger spread of results than usual, with more teams finishing with 0-1 wins on the low end and 0-1 losses on the high end. If the high end is particularly extreme, we could see even more 6 win teams fail to move on than last year, which is pretty difficult to accept as normal.