Share

Statsgasm Episode 4: Best Picture Nomination Voting Simulation (Pt. 2)

[Note: Part 1, posted last night, gave us an intro to the simulation methodology]

by Marshall Flores

With that out of the way, let’s begin our simulation. “Hold on to your butts.” I’m going to divide the entire BP nomination process into three rounds, each round consisting of multiple steps.

Round 1: Initial Allocation, Victory Threshold, and the Surplus Rule

We begin the first round of voting by separating the 505 top 10 lists I’ve compiled into piles of films receiving #1 votes. It turns out that although 357 films received votes from this sample of lists, only 80 films received #1 votes. The top 20 leaders in terms of #1 votes are:

top20_1votes

After we sort the ballots by their #1s, we then have to perform a few calculations. First, we’ll determine the initial victory threshold, the level of #1 support a film needs to get BP nominated off-the-bat. The victory threshold is approximately 9.1% of the total ballots cast (which is then rounded up to the nearest whole number), so 0.091 * 505 = 46 #1 votes.

Another number we need to determine is called the surplus threshold, which relates to a feature unique to IRV called the surplus rule. I consider the surplus rule an incentive for voters to consider both consensus and passion. The underlying idea is that if there is a strong degree of consensus on what are the #1 films, then other films that have support down the line on these ballots will benefit to a certain extent as well. In other words, the surplus rule is a potential reward for those voters who agree on a #1 film to a significant amount, by giving an additional degree of consideration to their 2nd or 3rd choices.

How does the surplus rule work? First, we have calculate the surplus threshold that a film needs to even qualify for the rule. This threshold is approximately 11% of the total ballots (and again, we round up), so 0.11 * 505 = 56 #1 votes. Note that any film meeting this level also meets the victory threshold, and as a result, *automatically* gets nominated for BP! As we can see in the table above, 12 Years and Slave and Gravity both meet the surplus threshold. So not only do we have two films with which the surplus rule applies, but we also have our first two BP nominees!

Next, and this is the wonkiest aspect of the surplus rule, we reallocate the votes of 12 Years a Slave and Gravity received to the #2 or #3 choices on those ballots at a reduced weight. This weight is calculated by determining how much each film surpassed the victory threshold, and then dividing these surplus #1 votes by the total amount #1 votes each film received.

Quite a mouthful, huh? Well let’s compute the vote weights in order to get a better idea. In the case of 12 Years, we have 69 #1 votes, which is 23 votes higher than the 46 vote victory threshold. As a result, the #2 choices on those ballots which listed 12 Years as #1 will receive 23 / 69 = 0.33 of a vote. Calculating along similar lines, the #2 choices on those ballots listing Gravity at #1 will obtain 19 / 65 = 0.29 of a vote.

Round 2: Allocating Surplus + Elimination of Films

We then start Round 2 by distributing the surplus votes. Our updated Top 20 below:

round2surplus

Although many films in our Top 20 benefited from the surplus rule, it’s not enough for any of them to meet the victory threshold. We now have to start eliminating films from contention. The elimination threshold is 1% of the total votes cast, or 0.01 * 505 = 6 votes. Films with vote totals below this level are removed from the BP equation and their votes will be redistributed to the next highest ranked film still in contention at full weight. This rule does give an incentive to voters in supporting any personal long shot favorites without wasting their ballots.

At this point, only 19 of 80 films are above the elimination threshold, so we eliminate 61 films from competition and attempt to transfer their votes. If a vote from an eliminated ballot cannot be reallocated, i.e. all the other ranked films on the ballot are no longer in contention, then the vote gets discarded and the total number of ballots in play is reduced. We end up discarding 12 ballots entirely, lowering our total vote count from 505 to 493.

Round 3: Allocating Eliminated Votes, 5% Nomination Threshold

We now enter the 3rd and final round of voting in our simulation. At this point, a film no longer needs to hit the 46 vote victory threshold to get nominated. Now, it just needs to garner at least 5% support of the ballots still in play, or 0.05 * 493 = 25 votes. Once the ballots eliminated in Round 2 are redistributed, any and all films receiving at least 25 votes *will* be BP nominated, while the rest are not. The nomination process then concludes.

Our updated and final results:

round3

So our simulation results in an 8 film BP lineup!

12 Years a Slave
The Act of Killing
Before Midnight
Blue Is the Warmest Color
Gravity
Her
Inside Llewyn Davis
Leviathan

All 8 films were in that Top 20 table that was ranked using a weighted system, but it’s very clear that #1 votes are paramount in obtaining a nomination. For example, Leviathan, which received about 50 total mentions less than either Frances Ha or American Hustle, ended up being BP nominated due to having more #1 votes than Frances Ha and Hustle *combined*. Practically speaking, it seems that a film needs to have been initially listed as the first choice on at least 3.5-4% of all ballots in order to have a shot at getting nominated.

These results are in line with other voting simulations that have been conducted by Steve Pond and others. When I ran this simulation last year (with just 207 Top 10 lists), I also arrived at 8 nominees. I can only speculate on how AMPAS has been able to come up with 9 BP nominees the past couple of years, but my intuition has come up with a couple of possible reasons. The first is that AMPAS has around 6000 voters – a much larger population of votes than the 500 in this simulation and others. The second – since voters are *not* required to rank additional films beyond their #1 choice, I would guess that there’s a high number of discarded ballots in Round 2. Discarding ballots ends up *lowering* the 5% threshold a film needs to reach in Round 3 to get nominated.

In the end, we ended up with 8 BP nominees, despite there being many possibilities for BP this year due to a strong overall crop of films. I would suspect that many would be pleased with a lineup as diverse as this one. AMPAS probably won’t be as diverse or bold with its lineup, but hey, we can all dream, right? :-)

An overall recap of the 3 rounds in our voting simulation can be found can be found here

This concludes this week’s very special (and long!) episode of Statsgasm. The process of nominating films for Oscar is a pretty involving and complex one, and arguments can certainly be made in favor of scrapping the current system and reverting back to a simpler one. But for now, instant-runoff voting is the name of the game.

As usual, feel free to ask any questions in the comments below, or contact me at marshall(dot)flores(at)gmail(dot)com and on Twitter at @IPreferPi314. Next week on Statsgasm we will return our focus to AD’s prediction models.

A very happy New Year to you all!

27 Comments on this Post

  1. HATE the weighted ballot.

    How about Most Votes. Period?

    Looking the Number One’s chart. There looks to be a natural cutoff between BLUE IS THE WARMEST COLOR and SHORT TERM 12. There is a tight bunch around BLUE and then a decent sized gap down to SHORT.

    Voila! 8 nominees. The end. Period. Easy.

  2. Voila! 8 nominees. The end. Period. Easy.

    As the results show, BLUE very barely meets the 5% needed to get nominated, its gain from the surplus votes being the deciding factor. So, no, it’s not really “easy” until all the votes are accounted for. :-)

  3. rufussondheim

    As implied in the other thread, I think AMPAS ends up with more films simply because they consider fewer films than do bloggers and critics when compiling their ballots. This leads to few “wasted” percentages. And with more scoring ability, that ninth film slides in.

  4. I think AMPAS ends up with more films simply because they consider fewer films than do bloggers and critics when compiling their ballots

    As I indicate in the very first paragraph in this post, the critics/bloggers gave votes to 357 films on these 500+ lists, but only 80 films received #1 votes. I don’t think that’s too far out of line with what actually happens in reality.

  5. Profile photo of Ryan Adams

    Since I can’t do the math and I’m an optimist I will always feel that if 10 nominees are “possible” then I’m justified in putting 10 nominees on my wish list.

    These are my 10 ‘gut instinct’ predictions that I put together 2 days ago before I saw Marshall’s latest installment.

    12 Years a Slave
    American Hustle
    Gravity
    Her
    Inside Llewyn Davis
    The Wolf of Wall Street
    Captain Phillips
    Nebraska
    Fruitvale Station
    Blue Jasmine (or Philomena) (or Saving Mr Banks)

    Those predictions differ from my own personal Top 20
    (films in English, ranking tentative, floating, not yet finalized):

    1. Inside Llewyn Davis
    2. 12 Years a Slave
    3. Her
    4. The Wolf of Wall Street
    5. Fruitvale Station
    6. Gravity
    7. Captain Phillips
    8. Nebraska
    9. Mud
    10. Before Midnight
    11. Frances Ha
    12. Short Term 12
    13. American Hustle
    14. The Counselor
    15. Enough Said
    16. All Is Lost
    17. Upstream Color
    18. The Place Beyond the Pines
    19. Blue Jasmine
    20. The Spectacular Now

    (I haven’t seen Dallas Buyers Club or Philomena yet)

    I’ll have different list for international films and documentaries — so don’t yell at me!

    It just makes no sense at this point for me to insert docs and FLF’s in my rankings because there are too many I haven’t yet seen.

  6. My bad, Marshall. Didn’t look at Part 1 close enough.

    STILL, you get a natural cut-off just looking at the Number Ones from the first round:

    12 YEARS
    GRAVITY
    LLEWYN
    HER
    BEFORE MIDNIGHT
    ACT OF KILLING
    LEVIATHAN

    All 7 have 20 or more #1s. There’s a gap down to 16 votes for BLUE IS THE WARMEST COLOR and as large a gap down to SHORT TERM 12.

    So, depending on your formula with percentages, you would either have 7 Nominees or 8.

    Simple. None of this IRV B.S.

    I hate the Oscar formula for much the same reason I hate the Electoral College. The most votes should win. Period.

  7. Bryce Forestieri

    Seeing NEW WORLD, BROKEN CIRCLE BREAKDOWN, and IN THE HOUSE today. I’m aiming to have my list done by Oscar nominations.

    LOL LEVIATHAN LOL

  8. Marshall,

    Don’t you just love the surplus rule? For best picture, it’s not so bad, but when you go into the other categories yikes.

    I still can foresee 10 nominees. From your list American Hustle and Wolf of Wall Street didn’t do well, probably because of their late release. Also, there will be titles that the industry loves while critics did not (and vice versa); think Blind Side.

    And I don’t think that the sample size going from 500 ballots to 6000 ballots would affect that much. 500 is a decent sample size. (Here I am ignoring the differences between the two voters.)

    I agree with you that the first round results typically do not change drastically.

    Good job.

  9. I still can foresee 10 nominees. From your list American Hustle and Wolf of Wall Street didn’t do well, probably because of their late release. Also, there will be titles that the industry loves while critics did not (and vice versa); think Blind Side.

    Although I certainly believe that there will be more BP nominees than less (which implicitly means that I absolutely do not buy the premise that a stronger year leads to fewer BP nominees), I don’t think release dates have any significant impact on these results.

    For instance, American Hustle was the 6th most mentioned film on the 505 lists I looked at with 131 votes, so it’s clear that it has been seen. Meanwhile, Her hasn’t even hit wide release yet and finished 4th in overall mentions and placed 3rd behind 12YAS and Gravity in #1 votes.

    Wolf is a different beast altogether – I think the divisive critical reception will certainly extend to the industry. But divisiveness also implies passion, and we must always keep in mind that the preferential ballot awards passion. I’m sure there are plenty of Scorsese partisans in AMPAS that will do their best to make sure Wolf gets in.

  10. Profile photo of Ryan Adams

    Yes, there’s going to be less than 1% of the Academy who will skip American Hustle.

    Thanks for verifying my own unscientific skepticism of the greater-the-year-fewer-the-nominees idea. That theory does not seem to stand up to commonsense scrutiny.

    I don’t place much stock in the almost-impossible-to-have-10-nominees theory either. In fact, a lot of what Steve Pond says makes little sense to me.

    The phrase “Mathematically Possible” has the word “possible” right there in the term.

  11. Could you imagine the uproar if this was AMPAS’ nominations?

  12. I don’t place much stock in the almost-impossible-to-have-10-nominees theory either. In fact, a lot of what Steve Pond says makes little sense to me.

    The phrase “Mathematically Possible” has the word “possible” right there in the term.

    Yeah, I myself will continue to predict 10 BP nominees as well, regardless of the strength of competition. As a practical matter I think it’s certainly difficult, and in general I’ll expect 8-9 films will make the lineup, but 10 is within the realm of possibility for this numbers guy.

  13. I believe the BP nomination ballot only allows five selections. And I believe (though could be corrected) that the ballot indicates the rank order. This would in all likelihood significantly reduced the number of films to those receiving the most publicity, industry buzz, for your consideration ads, critical acclaim and awards. The number of outliers is probably quite small. AMPAS members are not watching as many films as do critics.

  14. Marshall, I’m sure you know better than me, but I always thought that in Step 1 above, the surplus films were re-allocated down to the surplus threshold, not the victory threshold. Otherwise, what’s the point of the surplus rule?

    Imagine that Her had gotten nominated off the initial count with 48 votes, but did not meet the surplus requirement. If you reapportion 12YaS and Gravity votes down to the vote victory, then Her ends up with a highest count at the end of the nominations stage. I know it doesn’t mean anything, but that’d be… weird.

    It’s highly possible that I’m wrong, but I thought I’d just clarify.

    Also, I’d be interested to see what happens if you toggle the data so that 4/5 films meet the Surplus Threshold and again if you toggle it so that 1 or none meet it. Do the number of nominees change? I imagine that such an analysis would prove that it’s not necessarily the strength of the film year that determines the number of nominees, but the degree of consensus.

  15. In terms of Oscar:
    12 Years A Slave
    Gravity
    American Hustle
    Captain Phillips
    Nebraska

    I can’t see any of these five films missing at this point. From then on I’d say Her will make it. It’s the type of a film that benefits from passion voting. Inside Llwelyn Davis? I’m not sure, but lately the Academy has been very Coens-friendly. So that makes 7 films so far. Then comes August: Osage County. Harvey is pretty confident the film will get enough votes to make it into best picture. And yes, having Harvey Weinstein, Meryl Streep, Julia Roberts, George Clooney and a cast full of Oscar voters — maybe it will. The Academy always throws a not so well received film into the mix. And for number 9 I’ll leave Philomena. But what about Blue is the Warmest Color? Could it benefit from the passion voting? Could it make it into picture, directing, maybe even actress?

  16. I had more or less a vague idea about how Best Picture nominating process works, but thanks Marshall for making it very clear.

    Since I, myself, am into number crunching, I can very confidently say that going by how this whole process works, there are 5 films that are absolute certainty in Best Picture line-up:

    12 Years A Slave
    Gravity
    Her
    American Hustle
    The Wolf of Wall Street

    Why ? Because all the above 5 films will have their passionate supporters and hence will receive sufficient No.1 votes. I also feel, two more films, which you can include in the above list is Nebraska and Inside Llewyn Davis, though not as much certain as above 5 but still certain enough. That makes it 7.

    Captain Phillips may not receive many No.1 votes, but it will be on many ballots, so make that No.8.

    So, in case there are 8 films nominated this year [which I think is highly probable], it’s very clear which 8 films they are going to be.

    If, at all, 9 films get nominated, it’s a fight between Saving Mr. Banks, Dallas Buyers Club, Philomena and (surprise, surprise) Before Midnight.

    I say Before Midnight because it is going to be No.1 on several ballots. Now whether, that “several” will be enough, is highly doubtful. But I will still remain hopeful.

    Right now, I feel Saving Mr. Banks place in Best Picture line-up remains a question. Though, I feel among all the ones fighting for that last spot, its position remains the strongest. But please Academy, give us at least one surprise if not, more.

  17. Bob Burns

    so…. what time are the PGA noms announced?

  18. Hector Delgado, Jr.

    Nice Piece, though Leviathon and The Act of Killing are not eligible for the Best Picture Oscar, the other 6 feature choices which would be a pleasant surprise on nominations morning. Before Midnight and Blue Is the Warmest Color would be two choices that would elicit gasps from the crowd that morning.

    My predictions, in a strong year for film – eight nominees

    12 Years a Slave
    American Hustle
    Dallas Buyers Club
    Gravity
    Her
    Inside Llewyn Davis
    Nebraska
    Saving Mr. Banks

  19. rufussondheim

    Watched Frances Ha last night, at times it was squirmworthy at how much she was sabotaging her own life. Not because it was bad filmmaking, but because it was so relatable. This is really a special film.

  20. Profile photo of Ryan Adams

    The only reason Frances Ha missed being in my personal Top 10 was because the 10 movies I had to put before it are all so spectacular.

  21. Marshall, I’m sure you know better than me, but I always thought that in Step 1 above, the surplus films were re-allocated down to the surplus threshold, not the victory threshold. Otherwise, what’s the point of the surplus rule?

    Hi Evan, thanks for the question. With regards to the surplus rule, the vote level used in calculating the vote weights is indeed the 9.1% victory threshold, not the 11% surplus threshold. This is basically done to give the surplus votes some actual weight even if they aren’t whole votes. If the surplus threshold was used for calculations instead, we’d be looking at vote weights of 0.19 and 0.13 for 12YAS and Gravity, respectively, almost halving their effect.

    Again, my interpretation of the surplus rule is that it acts as an incentive for voters to rally around a #1 film so that their 2nd choices can benefit. Like any incentive, the potential benefit needs to be kinda significant.

    Imagine that Her had gotten nominated off the initial count with 48 votes, but did not meet the surplus requirement. If you reapportion 12YaS and Gravity votes down to the vote victory, then Her ends up with a highest count at the end of the nominations stage. I know it doesn’t mean anything, but that’d be… weird.

    If Her did obtain 48 #1 votes initially, it would’ve been nominated off-the-bat, effectively removing it from the competition. So it wouldn’t benefit from the reallocation of 12YAS’s and Gravity’s surplus – if it were #2 on those ballots, the #3 choices would receive the surplus instead. So yes, your scenario would be weird, but it also isn’t possible. :-)

  22. steve50

    This is really a special film. (Frances Ha)

    +1 Did you see Llewyn Davis yet, Rufus? There’s a shared chromosome between these two.

    I hope your stats are right, Marshall, because this is the most solid line-up in years. I’m just waiting to see what stinker will be thrown in the mix, or which of the most-worthy will be left off.

  23. I just realized that if the cast and crew from August: Osage County all vote for it as #1, it’d be pretty much in.

  24. This is fascinating reading, Marshall. I finally understand this convoluted voting process! I think it bodes well for PGA-ignored Inside Llewyn Davis, which has such passionate support and is sure to receive a high number of #1 votes. I would love to think Blue is the Warmest Colour might have enough passionate support to surprise, but I’ll try not to fool myself into thinking AMPAS has as good taste as critics.

  25. Marshall, under my scenario, I know Her wouldn’t get surplus votes. But it still would have 48 #1 votes and 12YaS and Gravity would have 46 post-reapportionment.

    But thanks for explaining the reasoning- your explanation about diminishing their weight if the surplus threshold is used makes sense.

    And nice analysis, of course.

  26. Ah, okay Evan, I get what you’re trying to say now. Yes, technically the reapportioning of the surplus votes does drop Gravity and 12YAS to 46 votes, and in this case, Her ultimately does end up with more votes at the end of Round 3. So it does look weird at first glance.

    But remember the nomination process isn’t a zero sum game, it’s just about crossing a certain finish line. Once a film breaches the required vote support levels, it’s in. How big or small the margin is doesn’t matter.

    And thanks! :-)

  27. John Samson

    I Love these stats and everything but does anyone think this is even possible. There is no way American Hustle does not get a nod.

Leave a Comment

Warning: Do not abuse your right to comment here. You will be deleted.