Impeachments
Would you like to react to this message? Create an account in a few clicks or log in to continue.

Structural Advantage for Group A in New ORCS Pairing System

+8
TheGhostofChaseMichael
TheRealMockProdigy
WaltzingwithBashir
happygolucky
GameCockMock
Pacificus
lookatmeguo
cfredricks
12 posters
Go down
avatar
cfredricks
Posts : 6
Reputation : 56
Join date : 2019-12-08

Structural Advantage for Group A in New ORCS Pairing System Empty Structural Advantage for Group A in New ORCS Pairing System

Sat Feb 29, 2020 6:39 pm
Message reputation : 88% (8 votes)
Brief Summary
I built a model to simulate the new ORCS pairing system. By running a Monte Carlo simulation, I discovered an apparent structural advantage to being in Group A. Basically, for any given team, moving them from Group D to Group A (but keeping them equally strong) will cause them to have a significantly easier schedule. In my model, moving a team from Group D to Group A increased their chances of making it to Nationals by ~5%. That is, if they would have a 15% of making it to Nationals from Group D, they would have a 20% chance of making it from Group A.

I was going to put all my analysis here, but the word processing in this text box is annoying, and I can't find a way to embed images. So if you want to read my analysis (and see some cool graphs that capture the problem), check out the PDF attached. I have also attached my ORCS simulator Excel model to this post, for anyone who wants to see it or play around with it.

I'm interested what y'all think of this. I don't think the advantage is intentional at all on AMTA's part, but it effectively rewards teams in Group A for past success, and makes it harder for teams in Group D to break through to Nationals for the first time.

Attached

  • Structural Advantage for Group A in New ORCS Pairing System (.pdf)
  • ORCS Simulator (.xlsm)


(Also, here is a Google Drive link if the download mechanism isn't working on here.)
https://drive.google.com/drive/folders/1LvTEbo8FUEDuBAUyLQ_F8aZ8diwvcWRh?usp=sharing
Attachments
Structural Advantage for Group A in New ORCS Pairing System Attachment
Structural Advantage for Group A in New ORCS Pairing System.pdf You don't have permission to download attachments.(554 Kb) Downloaded 64 times
Structural Advantage for Group A in New ORCS Pairing System Attachment
ORCS Simulator (Basic).xlsm You don't have permission to download attachments.(64 Kb) Downloaded 69 times
avatar
lookatmeguo
Posts : 7
Reputation : 24
Join date : 2019-11-20

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sat Feb 29, 2020 8:46 pm
Message reputation : 100% (1 vote)
This is really interesting, and I'm glad there are others looking at this from a data-driven perspective (even if it is 'fake' data Laughing ).

My initial reaction/follow-up question is this: How, if at all, does this variation in grouping affect who wins bids? Your analysis speaks to the variation in strength of schedule faced by a team, but not so much to the strengths of the six teams that ultimately win bids. I assume, from a normative perspective, that the desired outcome is the six teams of highest strength winning bids, regardless of their group affiliation. The Results Summary tab in your linked Excel document shows the 6 strongest teams making it out in this simulation. How often does that happen in other trials?

To take it a step further, we can define a measure of efficiency loss as the sum of squared distances between the strengths of the bid winners in an "efficient" outcome (i.e., {1,2,3,4,5,6}) and the strengths of the actual bid winners. If the ABCD system yields a lower expected efficiency loss than the old system, I would think that the variation in strength of schedule that you found is a tradeoff worth making.

EDIT: Another question that just occurred to me is whether it's realistic to model probability of winning as a function of group affiliation instead of as a function of realized team strength. I imagine the modeling is a lot easier when you have exogenous Group v. Group win probabilities, but imagine you have a A v. D matchup where the realized strengths of the teams (respectively) are 50 and 50. Under your model, the 50 strength team in the A Group has an 80% chance of winning each ballot, but I think it's more realistic that the probability should be closer to 50% since the teams are of equal strength.

Additionally, I suspect this may be a confounding factor in the variation you find in percentage chance of a 50 strength team winning a bid based on their Group (20.2% if A, 14.4% if D, according to page 4 of your pdf). The 50 strength team in A has, under your exogenous win rates, a 65% chance (on average) of winning each ballot across all four trials, while the corresponding average probability for the 50 strength team in D is only 35%. I think this could explain, at least in part, why the disparity you find is so large.
avatar
cfredricks
Posts : 6
Reputation : 56
Join date : 2019-12-08

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sat Feb 29, 2020 9:51 pm
Message reputation : 100% (1 vote)
lookatmeguo wrote:This is really interesting, and I'm glad there are others looking at this from a data-driven perspective (even if it is 'fake' data Laughing ).

My initial reaction/follow-up question is this: How, if at all, does this variation in grouping affect who wins bids? Your analysis speaks to the variation in strength of schedule faced by a team, but not so much to the strengths of the six teams that ultimately win bids. I assume, from a normative perspective, that the desired outcome is the six teams of highest strength winning bids, regardless of their group affiliation. The Results Summary tab in your linked Excel document shows the 6 strongest teams making it out in this simulation. How often does that happen in other trials?

Thanks for reading my analysis! You raise some good points. As for the question "How does the variation in grouping affects who earns bids?" it does make it more likely for a team in Group A to earn a bid, as opposed to an equal strength team in Group D. But the more basic point you're raising is valid. This new system very well might be better at identifying the top six teams than the old system. (Although, I think there is necessarily some efficiency lost with the Group A advantage. Teams that are top-six in strength in Group D will not earn bids as often as they "should.")

It does raise a deeper, philosophical question though. Let's say that the new system is better (in terms of identifying the top six teams) but less fair (in terms of giving some teams an advantage coming in, based on TPR. Do we prefer a system that is better but less fair?

lookatmeguo wrote:To take it a step further, we can define a measure of efficiency loss as the sum of squared distances between the strengths of the bid winners in an "efficient" outcome (i.e., {1,2,3,4,5,6}) and the strengths of the actual bid winners. If the ABCD system yields a lower expected efficiency loss than the old system, I would think that the variation in strength of schedule that you found is a tradeoff worth making.

Yeah if I modeled the old system this is definitely a question we could answer (whether the new system is more efficient). I suspect it is. I might get around to doing that later.

lookatmeguo wrote:EDIT: Another question that just occurred to me is whether it's realistic to model probability of winning as a function of group affiliation instead of as a function of realized team strength. I imagine the modeling is a lot easier when you have exogenous Group v. Group win probabilities, but imagine you have a A v. D matchup where the realized strengths of the teams (respectively) are 50 and 50. Under your model, the 50 strength team in the A Group has an 80% chance of winning each ballot, but I think it's more realistic that the probability should be closer to 50% since the teams are of equal strength.

Additionally, I suspect this may be a confounding factor in the variation you find in percentage chance of a 50 strength team winning a bid based on their Group (20.2% if A, 14.4% if D, according to page 4 of your pdf). The 50 strength team in A has, under your exogenous win rates, a 65% chance (on average) of winning each ballot across all four trials, while the corresponding average probability for the 50 strength team in D is only 35%. I think this could explain, at least in part, why the disparity you find is so large.

Yeah I think I wasn't clear on this. A 50-strength team has a 50% chance of winning a ballot, regardless of which group they are in. The "group win probabilities" are on average. Every team is then applied an "individual strength adjustment" between -20% and +20%. So if a team in Group A gets a randomly-assigned strength adjustment of -15%, then their true strength is 50% (65% based on their group, minus the 15% individual adjustment). A team in Group D that gets a +15% adjustment will also end up with a true strength of 50% (35% baseline because of Group D plus the 15%). So a matchup between the two teams would be a coin flip.

You're right that if I did it that way it would definitely mess up my results. But the way I did it, that shouldn't be a factor.
Pacificus
Pacificus
Posts : 46
Reputation : 48
Join date : 2019-12-18

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 12:38 am
Have you sent this to Bernstein and the TAC? If not, you should - although I know he lurks here.

I'm not a data guy but if I understand the effect being identified here correctly: Teams at the bottom of each bracket are shielded from the hardest teams in every other bracket by virtue of power matching across brackets. This benefits the worst teams in the A bracket, because the worst team in the A bracket will get to hit the worst teams in the B, C, and D brackets thanks to power matching. On average, this benefits the teams at the bottom of the higher tiers, because they are less likely to have to face the best teams in a lower tier, and more likely to face the worst.

Am I understanding this correctly? It makes intuitive sense now that I think about it.

As for whether it is better at picking the top six than the old system - whether it is or not, I'd rather compete in a system where random unfairness can hit anybody than a system which affords substantial systematic benefits to certain programs.
avatar
lookatmeguo
Posts : 7
Reputation : 24
Join date : 2019-11-20

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 1:29 am
Message reputation : 100% (1 vote)
Pacificus wrote:As for whether it is better at picking the top six than the old system - whether it is or not, I'd rather compete in a system where random unfairness can hit anybody than a system which affords substantial systematic benefits to certain programs.

I disagree. In my view, the purpose of the Regionals-ORCS-NCT progression is to identify the best team in the nation. In a hypothetical world, every team would be ranked by strength, where 1 is the strongest team, then 2, then 3, etc. In this world, a team would defeat all teams that have a larger rank number (e.g., team 2 would beat everybody except team 1). Teams 1-192 would make ORCS, 1-48 would make NCT, and 1 would win Nationals. (In more math-y terms: (X,R) is a totally ordered set where X is the set of teams in AMTA and R is a relation on X where aRb means team a defeats team b.)

Obviously, this isn't possible in reality. But, the mechanisms in place should be ones that effect outcomes closest to this ideal (on average/in expectation). Assuming that the ABCD system gets closer to the six best teams getting bids than the old system does, then ensuring that random unfairness that could hit anybody by preserving the old system would be promoting equity in pursuit of the wrong goal.

Additionally, I want to bring up one more hypothetical. Assume that the ABCD system gets "closer" to identifying the six best teams than the old system does. Now, let's say your team is new and unranked in TPR, but you're actually the sixth strongest team at your ORCS. Even if the ABCD system has systematic benefits that help the programs with low TPR numbers (Miami, Yale, UVA, etc.), you'd rather compete under the ABCD system because, as assumed, the ABCD system is more likely to identify you as a top-6 team than the old system.

For me, the ultimate question is whether the ABCD system does a better job of identifying the six strongest teams at an ORCS compared to the old system. I haven't run simulations to that end, but I'd love to see results if somebody else does.
GameCockMock
GameCockMock
Posts : 16
Reputation : 29
Join date : 2019-11-20

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 12:25 pm
Message reputation : 100% (2 votes)
This was really cool! When they announced the new pairing 
system I went through last year's ORCS to calculate what the matchup-by-matchup win rates would have been, had the teams been grouped. You can find the data here if you want to run anything again with those numbers, but your estimates weren't super far off. 

You've convinced me there's an effect here, and it seems it's caused by where your team falls within your tier, rather than which tier you're in. I may be misunderstanding, but being the best team in tier A is not an advantage over being the best team in tier D. However, either of those positions is disadvantaged compared to being lower ranked within either tier. Of course, a team with "true strength" in the top 6 at the tournament is way more likely to be in the #1 spot within C or D tier, because there probably won't be any other top "true strength" teams there with you (less than 5% of C & D tier moved on last year when pairing was indifferent to TPR rank). But being top 6 in true strength in tier A could have you fall almost anywhere within your tier, so on average your schedule will be easier there.

I think this set-up may still be preferable to the old one, depending on the size of the effect in actuality. I don't see a reason we should prioritize eliminating all structural unfairness IF the result is increased overall unfairness, just because its randomly applied unfairness. I don't know whether that's true of the old system, but it seems it might be. If the effect size of the lucky/unlucky scheduling is equal between a random and a structural option, I definitely agree the random option is better in that case. But because A & B tiers were even more successful (last year with random pairing) than your estimates show, I think that may shrink the effect. The main attraction for me to this new system is the hope it will reduce absurdly high CS that practically prohibits a team from moving on. I'm more concerned about everyone being guaranteed a reasonable schedule, so long as any structural bias is slight and not existent when comparing tiers overall. I'd love to see data on how this would play out with last year's tier matchup win rates, and which system more consistently ends with the maximal "true strength."


Last edited by GameCockMock on Sun Mar 01, 2020 12:26 pm; edited 1 time in total (Reason for editing : link formatting)
happygolucky
happygolucky
Posts : 38
Reputation : 12
Join date : 2019-11-23

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 12:32 pm
Message reputation : 100% (1 vote)
This looks really interesting - can you help out a less Excel inclined mocker here, I'm trying to run a few simulations but it only shows the ###### sign. Am I missing a step?
avatar
WaltzingwithBashir
Posts : 9
Reputation : 2
Join date : 2019-12-30

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 12:55 pm
Message reputation : 100% (1 vote)
I too get a #NAME? error. It seems to be a formula issue with the XLOOKUP but I'll differ to the excel gurus.
Pacificus
Pacificus
Posts : 46
Reputation : 48
Join date : 2019-12-18

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 1:35 pm
Obviously, this isn't possible in reality. But, the mechanisms in place should be ones that effect outcomes closest to this ideal (on average/in expectation). Assuming that the ABCD system gets closer to the six best teams getting bids than the old system does, then ensuring that random unfairness that could hit anybody by preserving the old system would be promoting equity in pursuit of the wrong goal.

I guess I disagree on a philosophical level with this - consider the problem we are trying to solve: certain teams of high quality losing out due to random pairing against other very high quality teams, while certain low quality teams breeze through with an easy schedule.

Now consider the example of Weighted Partial Ballots, a system which has been around for a long time (in various forms) and which has always been able to "smooth out" this problem at any given tournament. Yet it has never gone anywhere, and AMTA never adopted it for ORCS and Regionals. Why not? I don't know the reasoning they have, but the best argument against WPB has always been that it seems to leave open the possibility of a team beating all their opponents and still not qualifying - which just seems unfair, even if all their opponents happened to be pretty bad. So the random unfairness persisted, because we did not want to violate the principle that a team which wins every round should place well at a tournament - and even though I bet there is a strong argument WPB would have done a better job of selecting true "top" teams.

I feel similarly about the effect identified here. Yes, it smooths out the random mismatching problem. But if it does so by systematically benefiting certain programs and harming others on the basis of a metric they really have no control over, isn't it violating the fundamental idea that everybody should have an even chance to win the tournament going into round one? I think that principle is important to any competition- who wants to compete in a competition that is inherently biased toward certain competitors? If we only cared about getting the top 196 teams to ORCS and the top 48 teams to nationals, we could probably cancel the actual competition and just go down the TPR rankings and pick 'em: 1-196 to ORCS, 1-48 to Nats. This would probably be pretty accurate, if we're being honest, in terms of determining the top teams. But it wouldn't be competitive at all.
avatar
TheRealMockProdigy
Posts : 82
Reputation : 99
Join date : 2019-11-25

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 2:39 pm

Now consider the example of Weighted Partial Ballots, a system which has been around for a long time (in various forms) and which has always been able to "smooth out" this problem at any given tournament. Yet it has never gone anywhere, and AMTA never adopted it for ORCS and Regionals. Why not? I don't know the reasoning they have, but the best argument against WPB has always been that it seems to leave open the possibility of a team beating all their opponents and still not qualifying - which just seems unfair, even if all their opponents happened to be pretty bad. So the random unfairness persisted, because we did not want to violate the principle that a team which wins every round should place well at a tournament - and even though I bet there is a strong argument WPB would have done a better job of selecting true "top" teams.

This is not an accurate description of WPB, imo. WPB actually rewards you for hitting bad teams -- if you can run up big PDs, you get the "full" ballot. But if you have four rounds against good teams and they are all close, you don't get the "full" ballot. At the risk of exposing too much about myself, the one tournament I ever went to that used WPB, my team went 8-0 and came in third. In first was a team with an 8-0 record and a worse cs. They had hit easy teams and run up big PDs. More egregiously, the team in second was a team that went 4-4, and two of the ballots they lost were to my team.

More broadly, the big issue with WPB is that it assumes all judges score the same. Some judges use the full range of scores and allow for big PDs. Other judges use a narrower range and smaller PDs. But WPB acts as if the range is the same for all judges, which is problematic.
Pacificus
Pacificus
Posts : 46
Reputation : 48
Join date : 2019-12-18

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 3:03 pm
Message reputation : 100% (1 vote)
This is not an accurate description of WPB, imo. WPB actually rewards you for hitting bad teams -- if you can run up big PDs, you get the "full" ballot. But if you have four rounds against good teams and they are all close, you don't get the "full" ballot.

While I understand the specifics of WPB can be and often are pretty heavily adjusted (such as tinkering with what PD earns a "full" ballot) I think the bigger issue is the systematic unfairness. But even if one could assume the PD was always representative of how "big" of a win Team A got over Team B, the mere possibility that you could beat Team B and still end up ranking behind them (as you pointed out) goes against the spirit of the competition.

When this new system was announced, my gut reaction was "I don't like it". I thought about it for quite a while, trying to figure out why I didn't like it, but in the end, I was forced to admit to myself there did not appear to be any systematic bias built in, and I was just being a crusty old mock trial nerd set in my ways. It appeared to be designed quite fairly, and to solve the random-hard-schedule problem. But this analysis by OP seems to show, fairly conclusively, that this program builds in a bias for teams on the bottom end of the A bracket and so I find myself changing my mind again. The magnitude of the bias has yet to be determined. But I think it behooves AMTA to investigate and make appropriate adjustments.
avatar
lookatmeguo
Posts : 7
Reputation : 24
Join date : 2019-11-20

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 3:15 pm
WaltzingwithBashir wrote:I too get a #NAME? error. It seems to be a formula issue with the XLOOKUP but I'll differ to the excel gurus.

Yep, XLOOKUP is a very new function and I don't think it's backward compatible with older versions of Office. I get a similar error when I open the file on my computer.
Pacificus
Pacificus
Posts : 46
Reputation : 48
Join date : 2019-12-18

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 3:15 pm
I would also add that OP says the effect size could get larger, if the strength difference between groups is bigger:

In my model, it worked out to a pretty consistent advantage of about 5%. That is, moving a team from Group D to Group A would boost their chances of getting a bid by 5% (e.g. from 15% to 20%). But that effect could be even bigger if the strength difference between groups is larger.

I think there is a very strong likelihood that this is the case - as mentioned elsewhere teams in the C and D tiers almost never qualify to nationals, historically speaking - and I think the random variation year over year for most programs is probably a lot less than 20%. I think in the real world A tier teams beat D tier teams probably more like 90-95% of the time, not 80%. The disparity between the top 50 programs in the TPR rankings and those below 100 is much, much bigger than this model assumes.
avatar
cfredricks
Posts : 6
Reputation : 56
Join date : 2019-12-08

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 3:36 pm
Message reputation : 100% (3 votes)
Pacificus wrote:Have you sent this to Bernstein and the TAC? If not, you should - although I know he lurks here.

I have not. I suppose I could do so soon. One benefit of posting on here is that if anyone sees any errors or ways to improve my analysis, I can potentially implement those before sending this to TAC. So far, I don't think that there has been anything that people have pointed out that would change my overall conclusion—although implementing the data from GameCockMock could definitely help me better quantify how large the effect is.

Pacificus wrote:I'm not a data guy but if I understand the effect being identified here correctly: Teams at the bottom of each bracket are shielded from the hardest teams in every other bracket by virtue of power matching across brackets. This benefits the worst teams in the A bracket, because the worst team in the A bracket will get to hit the worst teams in the B, C, and D brackets thanks to power matching. On average, this benefits the teams at the bottom of the higher tiers, because they are less likely to have to face the best teams in a lower tier, and more likely to face the worst.

Am I understanding this correctly? It makes intuitive sense now that I think about it.

GameCockMock wrote:You've convinced me there's an effect here, and it seems it's caused by where your team falls within your tier, rather than which tier you're in. I may be misunderstanding, but being the best team in tier A is not an advantage over being the best team in tier D. However, either of those positions is disadvantaged compared to being lower ranked within either tier. Of course, a team with "true strength" in the top 6 at the tournament is way more likely to be in the #1 spot within C or D tier, because there probably won't be any other top "true strength" teams there with you (less than 5% of C & D tier moved on last year when pairing was indifferent to TPR rank). But being top 6 in true strength in tier A could have you fall almost anywhere within your tier, so on average your schedule will be easier there.

Both of these understandings of the effect are basically correct. But I do want to emphasize: there is a distinct benefit to being in Group A, no matter how strong your team is. This is basically for the reason that GameCockMock articulated. No matter how good your team is, you're more likely to find yourself (throughout the tournament) in the top spot in Group D than Group A. Even if you are Rhodes A, and you are the best team in the tournament, you would rather be in Group A, because if you make small mistakes or get a little unlucky in the first few rounds, that will likely be enough for you to fall to a much lower spot in the Group A standings. If you are in Group D, on the other hand, you'll likely still be in the D1 or D2 spot even after dropping a ballot or two. In other words, every team should, mathematically, want to be in Group A.

GameCockMock wrote:This was really cool! When they announced the new pairing system I went through last year's ORCS to calculate what the matchup-by-matchup win rates would have been, had the teams been grouped. You can find the data here  if you want to run anything again with those numbers, but your estimates weren't super far off.

But because A & B tiers were even more successful (last year with random pairing) than your estimates show, I think that may shrink the effect.

This is great data that I should be able to implement into my model. That will allow me to better quantify just how large the effect is. With my estimates, it resulted in a 5% increase in your bid chances moving from Group D to Group A. But I think you are incorrect about how your data will impact this number. If my understanding of the effect is accurate, then the bigger the disparity between Group D and Group A, the larger the effect. If all groups are equally strong, then no effect exists. But if Group A is wayyyy stronger, the advantage to being in Group A could be quite large.

happygolucky wrote: This looks really interesting - can you help out a less Excel inclined mocker here, I'm trying to run a few simulations but it only shows the ###### sign. Am I missing a step?

WaltzingwithBashir wrote: I too get a #NAME? error. It seems to be a formula issue with the XLOOKUP but I'll differ to the excel gurus

Oh no! I suspect this does have to with the XLOOKUP function. It is a newly-released formula, so if you haven't updated your copy of Excel or are running an older version, that is probably causing the issue. You should also make sure you are clicking yes on "Enable content" and/or "Enable macros". That is necessary for the "Run Simulation" button to work.

I hope these problems aren't too widespread.
Pacificus
Pacificus
Posts : 46
Reputation : 48
Join date : 2019-12-18

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 4:28 pm
If my understanding of the effect is accurate, then the bigger the disparity between Group D and Group A, the larger the effect. If all groups are equally strong, then no effect exists. But if Group A is wayyyy stronger, the advantage to being in Group A could be quite large.

Can you re-run the models with GameCockMock's data? It appears from his data that Tier A wins about 55% of rounds against Tier B,  77% of rounds against Tier C, and 89% of rounds against Tier D. If I'm reading the sheet correctly. That would give you the most accurate estimate of the real-world effect size.

I'm playing with it in the posted spreadsheet, but it doesn't appear to produce the graph your analysis posts.
avatar
cfredricks
Posts : 6
Reputation : 56
Join date : 2019-12-08

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 4:35 pm
Message reputation : 100% (2 votes)
Pacificus wrote:
If my understanding of the effect is accurate, then the bigger the disparity between Group D and Group A, the larger the effect. If all groups are equally strong, then no effect exists. But if Group A is wayyyy stronger, the advantage to being in Group A could be quite large.

Can you re-run the models with GameCockMock's data? It appears from his data that Tier A wins about 55% of rounds against Tier B,  77% of rounds against Tier C, and 89% of rounds against Tier D. If I'm reading the sheet correctly. That would give you the most accurate estimate of the real-world effect size.

I'm playing with it in the posted spreadsheet, but it doesn't appear to produce the graph your analysis posts.

Yes I will tonight. I performed the Monte Carlo stuff in some added sheets that aren't in the one I posted here (because they use formulas that are based on an Excel plug-in you have to download). I need to rework those sheets a little bit to work with the new data, and it will probably take my computer a long time to run the number of iterations that I need to smooth out the graphs. So I will probably have it run a large number of simulations overnight.

Based on a small batch that I just ran, though, the effect does appear to grow with the new numbers. Won't know for sure until I run the full thing.
avatar
TheGhostofChaseMichael
Posts : 12
Reputation : 16
Join date : 2019-11-19

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 8:13 pm
Message reputation : 0% (2 votes)
So here are some really big things I noticed about your model, and I think they may account for a large percentage of the "advantage" your data is showing for A-bracket teams. Your model basically bakes in a massive advantage for A-bracket teams from the beginning in the way that you calculate who will win each round. I think this (possibly more than anything else) accounts for the better performance of A teams in the final results. To be clear, this is an effect created by you statistical model not the new AMTA system.

When you are computing who will win each round, you compute the probability of a win for the P team (based on what I can glean from excel, which is kind of glitchy for me since I have an old version) as

(Probability based on group)+(P adjustment)-(D adjustment).

This fundamentally gives higher TPR teams an advantage in all rounds where the strength of the teams is roughly equal. Suppose, for instance, that we take an A team with a "strength" of 50 and a D team with a "strength" of 50 and we match them up. Then, intuitively, we would expect that the A team should win in 50% of cases (because they are the same strength). But on your model, the A team has a strength adjustment of -15 and the D team has a strength adjustment of +15. So, the A team's probability of winning will be 80%-15%+15%=80%.

In other words, the way you are calculating who wins gives an advantage to the higher TPR teams and makes them win more often (even when they shouldn't). This has noting to do with the AMTA pairings and everything to do with who you predict to win each match-up.

This suggests that some (if not all) of the advantage your graph shows is the result of your prediction algorithm rather than the AMTA system.

avatar
cfredricks
Posts : 6
Reputation : 56
Join date : 2019-12-08

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 8:29 pm
Message reputation : 100% (3 votes)
TheGhostofChaseMichael wrote:So here are some really big things I noticed about your model, and I think they may account for a large percentage of the "advantage" your data is showing for A-bracket teams. Your model basically bakes in a massive advantage for A-bracket teams from the beginning in the way that you calculate who will win each round. I think this (possibly more than anything else) accounts for the better performance of A teams in the final results. To be clear, this is an effect created by you statistical model not the new AMTA system.

When you are computing who will win each round, you compute the probability of a win for the P team (based on what I can glean from excel, which is kind of glitchy for me since I have an old version) as

(Probability based on group)+(P adjustment)-(D adjustment).

This fundamentally gives higher TPR teams an advantage in all rounds where the strength of the teams is roughly equal. Suppose, for instance, that we take an A team with a "strength" of 50 and a D team with a "strength" of 50 and we match them up. Then, intuitively, we would expect that the A team should win in 50% of cases (because they are the same strength). But on your model, the A team has a strength adjustment of -15 and the D team has a strength adjustment of +15. So, the A team's probability of winning will be 80%-15%+15%=80%.

In other words, the way you are calculating who wins gives an advantage to the higher TPR teams and makes them win more often (even when they shouldn't). This has noting to do with the AMTA pairings and everything to do with who you predict to win each match-up.

This suggests that some (if not all) of the advantage your graph shows is the result of your prediction algorithm rather than the AMTA system.


You're right in interpreting my formula, wrong in your algebra.

(Probability based on group) + (P adjustment) - (D adjustment).
(80) + (-15) - (15)
80 - 15 - 15
50

So yes, each team would have a 50% of winning the round. My formula is correct, and this is not what is generating the advantage I found.
avatar
TheGhostofChaseMichael
Posts : 12
Reputation : 16
Join date : 2019-11-19

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Sun Mar 01, 2020 9:07 pm
Message reputation : 100% (3 votes)
You are right. Embarrassing mistake for a math nerd. I looked too fast. Shocked
avatar
cfredricks
Posts : 6
Reputation : 56
Join date : 2019-12-08

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Mon Mar 02, 2020 5:30 pm
Message reputation : 100% (3 votes)
Pacificus wrote:Can you re-run the models with GameCockMock's data? It appears from his data that Tier A wins about 55% of rounds against Tier B,  77% of rounds against Tier C, and 89% of rounds against Tier D. If I'm reading the sheet correctly. That would give you the most accurate estimate of the real-world effect size.

Ok, I re-ran it with the real-life data and here are the results.

Structural Advantage for Group A in New ORCS Pairing System Orcs_b10

1) Group A still holds a significant advantage. Group B has a lesser advantage, followed by C and D. Groups C and Group D are basically identical in these graphs, partially due to GameCockMock's data (which showed that Group C is not very much better than Group D) and probably partially due to the issue described at the bottom of this post.

2) The size of the effect is larger than I had first estimated, especially for teams that are decently strong. The size of the impact is clearest to see between the strength ranges of 55 and 70. These are teams that have a decent shot at getting a bid but will need to have a strong tournament and maybe get a little lucky. See the graph below, which is zoomed in on that range.

Structural Advantage for Group A in New ORCS Pairing System Orcs_z10

3) Over this range, the benefit of moving from Group C to Group A is about 7%. For example, a team with strength 60 would get a bid 30% of the time if they were in Group C, and 37% of the time if they were in Group A. Because of how bad teams in Group D had to be to match GameCockMock's numbers, there were no Group D teams this strong. That is why Group D is not on this graph. But I suspect if there were a Group D team this strong, the disparity would be even wider, probably 8-10%.

4) The effect is smaller for worse teams. For teams with a strength between 43 and 53, the advantage was between 3% and 5%. This is intuitive. Teams that don't have a great chance of making it regardless aren't hurt as much. The disadvantage grows the better your team is.

One caveat: @lookatmeguo helped me identify a small issue related to Group D vs. Group A matchups that is slightly boosting Group D and slightly harming Group A. Essentially, in their head-to-head matchups, this error is causing Group A to only win 87.6% of ballots instead of 89%. In my mind, this probably explains why Group D is in line with Group C, instead of slightly below. I want to emphasize this, though: this issue is actually boosting Group D slightly and harming Group A slightly, which means that if we find a way to fix it, the size of the disparity should only grow. The Group A advantage shown in these graphs is in spite of the issue, not because of it.
avatar
Mocka Flocka
Posts : 9
Reputation : 48
Join date : 2019-11-18

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Tue Mar 03, 2020 1:35 am
Message reputation : 50% (2 votes)
homeboy is out here writing bitcoin white papers on ORCS pairings
avatar
briefly_your_honor
Posts : 16
Reputation : 6
Join date : 2020-01-21
Location : "The Midlands"

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Wed Mar 04, 2020 5:46 pm
Additionally, I want to bring up one more hypothetical. Assume that the ABCD system gets "closer" to identifying the six best teams than the old system does. Now, let's say your team is new and unranked in TPR, but you're actually the sixth strongest team at your ORCS. Even if the ABCD system has systematic benefits that help the programs with low TPR numbers (Miami, Yale, UVA, etc.), you'd rather compete under the ABCD system because, as assumed, the ABCD system is more likely to identify you as a top-6 team than the old system.



^^^^

This point is why I think this new system is going to work well to identify the closest there is to the top 6 teams for each ORCS. If you're an unranked team but currently very strong that means, inherently, that your team has little or no experience with the level of competition in an ORCS. But, if you are a middle pack team (say a bottom of B top of C group team) it is likely you have some program experiences with ORCS, likely recently based on the rankings that are mostly making up those groupings, and so you are better equipped for the stress and rigor of what's about to happen. New programs or just inexperienced teams can do ok in invitationals but when some of the more technical AMTA issues start to crop up they are usually unprepared/not familiar with AMTA's rules enough. That alone can create an advantage for teams that consistently go to an ORCS or higher. This new system makes is more likely that if you are an unranked D group team you won't wind up hitting 2 or 3 teams with years of experience and structural advantage, so your chances are better that the actual abilities of the team (and not just their knowledge of the limits of the AMTA rules) will be the deciding factor in a round.
avatar
Birch
Posts : 11
Reputation : 9
Join date : 2019-12-04

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Mon Mar 09, 2020 10:25 pm
Message reputation : 100% (1 vote)
So, I've started trying to figure out the predictiveness of brackets. I'm still inputting a lot of information, but so far I've already entered data for Lancaster and Santa Monica.

Since this simulation relied on predictions regarding how many ballots particular brackets would take off of each other, I've summarized one of my findings: the actual resulting ballot percentages. Here is the chart. I will update it in the coming days.

Structural Advantage for Group A in New ORCS Pairing System Summar10

This chart should be read as "When the bracket in the left column hits the bracket on the top, what percentage of ballots do they take".

Notable is that within the current data set, B did better against D then A did, and A did better against C than they did against D. Otherwise, the expected trends seem to hold. Given the limited data set (12 rounds for each box), it's possible this is just a deviation from the true mean. Or maybe I made an excel error.
avatar
adamsel
Posts : 51
Reputation : 70
Join date : 2019-11-19

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Mon Mar 09, 2020 10:35 pm
Message reputation : 100% (2 votes)
Birch, I did something similar but got different numbers. More notably, I had B bracket beating A bracket 55% (32.5 ballots to B and 27.5 to A). I could be wrong, but there's also a possibility that you inputted some numbers wrong

Edit: I see you only did Lancaster and Santa Monica. I did all 5, below is what I found

Structural Advantage for Group A in New ORCS Pairing System Screen11
Sponsored content

Structural Advantage for Group A in New ORCS Pairing System Empty Re: Structural Advantage for Group A in New ORCS Pairing System

Back to top
Permissions in this forum:
You cannot reply to topics in this forum