The election season is fascinating for math geeks, because it’s full of numbers, and numbers about numbers, and numbers about numbers about numbers.
Now, all the people who did statistics will find this boring, but I know a lot of people don’t really have a stats background, and some of this stuff is interesting and worth knowing more about, so I’m gonna ramble a bit on what statistics mean and why they work the ways they do.
Statistics is the art of trying to get a good guess as to the overall traits of a group without having to examine everything in the group. This is useful, because sometimes you can’t examine everything. For one thing, it may take far too long. For another, if you’re trying to find out how strong a material is, you really want a way that lets you guess how much force it would take to break a given object, rather than breaking everything to find out how strong it was. Consider bridges; building a bridge, loading it up until it collapses, and then writing up a sign that says “the bridge that was here would have had a weight limit of 40 tons” does not actually help you very much.
Statistics are still guesses. There are two common ways this uncertainty is expressed. One is a degree of confidence; we are X% sure that the value of something is at least Y. The other is what’s called a margin of error. You’ll see this a lot in polling; pollsters will say that 44% of people support X, and 39% oppose X, and the margin of error is 3. In fact, a margin of error is a fancy thing on top of a confidence interval; what “3 margin of error” usually means is “we are 95% sure that the true values are within 3% of the reported values”. It’s usually 95, says Wikipedia; it could easily be other values, but usually people pick 90 or higher. The margin of error for a given level of confidence is sometimes called a confidence interval.
In general, the higher the confidence you want, the broader the margin of error. The margin of error will go down as your sample size increases, and up as it decreases. So a survey of only a small number of people will tend to have a larger margin of error.
So when a poll says that candidate A has 49% of the vote, and candidate B has 47, with a 2 margin of error, that means that they are about 95% certain that the actual values are somewhere between 47-51% for A, and 45-49% for B. And that, in turn, means that it is entirely possible that B is actually more popular, by a small margin. (And indeed, it’s possible that the real margin is outside that range; it’s just not very likely.)
In the US presidential elections, considering the vast majority of states which assign all electoral college votes to a single candidate, it may seem strange that predicted electoral college totals are often not the result of any combination of states voting for a candidate. Rather than predicting each state, then summing the totals, pundits are more likely to assign weighted values for each state. So, Nate Silver assigns a 71% chance of Iowa going to Obama this year. What this means is that he’ll probably assign about 71% of 6 votes to Obama in his predicted electoral college vote total, and 29% of them to Romney. This is why you have predictions like his current 294.6:243.4; obviously, neither candidate is getting .6 of an electoral college vote, but the predicted average is in there. The margin of error, however, is huge; the same site gives about a 72.9% chance for Obama to win. That means a 27.1% chance that his total is 25.6 votes lower than expected, so a 90% or 95% confidence interval is presumably somewhat further out.
As a general rule, human intuition about statistics is dismal; we tend to heavily overrate the accuracy of small samples, and most people simply ignore “margin of error” values mentioned for polls. The other common mistake is to assume that the true values must be within the margin of error of every poll (which is obviously untrue; they should be outside the margin of error of about 1 poll in 20 usually), or that failure of polls to agree proves that one of them is biased.
There are probably 5-10 polls on the election occurring per day in the US, maybe more. Given that, you should expect that every two to four days, there will be at least one reasonably major poll where the “real” values are outside the margins of error listed for the poll.
Note also that polls, in general, are subject to systematic biases, which is one of the reasons that pundits tend to use multiple polls, and give them different weights. You can typically get a better value by comparing many polls than you can from any one poll. It’s probably not generally the case that the biases are particularly premeditated or malicious, but methodology will affect things. Polls done to cell phones will get you a very different set of people than polls to land lines. Polls will understate the votes of people who don’t have phones at all or don’t talk on them. (For instance, I know many autistic people who have consistently hung up on phone polls. Of course, that’s a tiny minority, and probably statistically insignificant.)
It is interesting to note that people are much more likely to think a poll or commentary site biased when it goes against their preferences. There are clearly some systematic biases floating around, but it’s hard to be sure which ones they are. Add in the general background flood of allegations that votes are being fixed, and you can have the argument over whether the polls didn’t represent the outcome due to bias in the polls or due to fraud.
You might want to know what happens when you try to use multiple samples to accumulate a better sample. Long story short: People make a ton of mistakes doing this. I actually don’t know how to do it correctly, but I am aware that simply averaging statistics does not give you better statistics.
So long story short: It’s a fancy word for guesses, but the guesses can be extremely good with careful attention to methodology, etcetera.
Comments [archived]
From: Dave Leppik
Date: 2012-10-30 14:46:44 -0500
Nate Silver’s forecast is not a sample, it’s a forecast, generated via a Monte Carlo simulation. That is, he took the poll numbers, weighted them, and did hundreds of simulations in which the probability of each precinct voting in a particular way matched the weighted poll numbers.
The statistical analysis of a Monte Carlo simulation is a lot trickier. It’s easy to run enough trials to eliminate margin of error. Given enough trials, the result is always the same. However, because it is a simulation, it is only as good as the data that goes into it. That’s where things get really tricky.
To know how accurate the model is, you need to know not only how good your assumptions are, but how each assumption is related to each other. That is, if the Southern states tend to vote together not out of random coincidence, but because of some hidden factor that the model doesn’t account for, the model will be inaccurate.
For example, the model assumes that Romney has a 65% chance of taking Florida, but only a 42% chance of taking Virginia. Those numbers are based on a weighted average of polls, and (as far as I know) they are thrown into the simulation as-is. That is, each trial rolls the dice independently on Florida and Virginia. Which means that if Romney takes Virginia, there’s still a 65% chance that he will take Florida.
However, in the real world, if Romney takes Virginia, there’s probably some reason that also effects Florida, so the chance of Romney (or Obama) taking both states is higher than the model predicts.
I don’t know how you estimate the reliability of a Monte Carlo simulation. If you have convergent variables (e.g. states which tend to vote together), it will underestimate the convergence— and fail to predict a landslide. If you have divergent variables (e.g. a pro-Obama outcome on the East Coast keeps Democrats at home on the West Coast), it will underestimate those too— and fail to predict a tight race.
Monte Carlo simulations can also be prone to “butterfly effect:” tiny errors in the model can be compounded, as their effects are multiplied together. This seems less likely in the Five Thirty Eight Forecast, since electoral votes are summed rather than multiplied.
From: DES
Date: 2012-10-31 11:30:36 -0500
I am 95% certain that either Obama or Romney will win.