CA 120: Understanding the polls

There is likely nothing more confusing to voters than the avalanche of polls that spill forth every election year. With what many consider to be America’s most crucial election in generations just days away,  this lengthy piece from pollster and election data analyst Paul Mitchell is a valuable guide to understanding how polls are conducted and presented to you, the voters. 

Going into the final weekend of the 2024 General Election, the presidential contest appears to be in a dead heat. The changing of the Democratic nominee, two conventions, two assassination attempts, one debate, countless speeches and town halls and podcasts, people eating cats and dogs and islands of trash in the ocean, and, frustratingly, nothing has pushed the contest out of the “toss-up” range.

The margins may be narrow, but there is widening set of tools to describe them – from polls to forecasts to prediction markets and early vote projections.

With days to go, we cannot predict who can win, but we can at unpack some of these metrics, and illustrate some of the strengths and weaknesses of different methods that will shape the political coverage until 4pm on Tuesday when we get our first election results from critical swing states on the East Coast.

The Polls
The presidential poll is celebrating its bicentennial this year, with the first recorded polling in the Andrew Jackson versus John Quincy Adams election in 1824. Back then our entire election system was different, but newspapers would conduct straw polls at local events, like a state fair, and report those results.

The polling industry matured over the next 100 years with things like magazines sending readers postcards to obtain completed surveys, and then later with Gallup producing the first statistically based polling in 1936.

The basic theory of polling is one of sampling.

When you are cooking, and you want to know if your soup has the right spicing, you take a little spoonful. You don’t need to eat the whole thing to determine if it is right – just a little will do. If it is a soup with corn and chicken in it, you may hunt around so the contents of your spoon has a little of everything before you taste it.

Polls work the same way. They use a small sample to try and determine the beliefs, opinions and attitudes of a full pot of voters. In a city council district race, they could survey 400 people and say with some confidence that the result will mirror the actual views of the 10,000 voters in that district.

Much like the chicken soup analogy, the 400 respondents do need to be close to matching the composition of the actual electorate – having the right number of key subgroups like age, ethnicity, partisanship, gender and other factors that are sometimes, but not always, used by pollsters, like, region, education and income.

While a pollster could get extra responses from key subgroups groups, like contacting 25 more Latinos to make the survey 425 respondents and a bit more Latino, that is unlikely to get all the subgroup percentages exactly right. Instead, pollsters do what is called “weighting” where they will add a small multiplier to the responses from groups that are underrepresented in their 400-person sample, and maybe place a little downward mathematical pressure on respondents who are overrepresented in the sample. At the end of the survey, because of this weighting, each 65+ white Democrat maybe counts for 90% of a respondent, while each Gen Z Independent is having their response count for 1.1 voters. After all the respondents are weighted, the result is put back together, giving a final result which should match the composition of the electorate.

With days to go, we cannot predict who can win, but we can at unpack some of these metrics, and illustrate some of the strengths and weaknesses of different methods that will shape the political coverage until 4pm on Tuesday when we get our first election results from critical swing states on the East Coast.

Once completed, each poll comes with a Margin of Error which can easily be calculated, and in this case of a 400 person poll in an area with 10,000 voters, it would be plus or minus 5%. If a result is 60% – 40%, that means that the candidate leading the poll is at 55%-65%, and their opponent is at 35%-45%. When polls are 49%-47% or 50%-48%, like we have seen so many times in this year’s presidential race, the poll is basically saying it has no idea. Go home and flip a coin.

But this margin of error isn’t the only possible flaw in the polling.

Sometimes the poll is incorrectly assuming what turnout will be and therefore it is weighting the poll incorrectly. Or a poll could be done in a race with a leading Vietnamese candidate, but isn’t including Vietnamese language callers. Or, maybe the call center is making a mistake in how they conduct the survey, like mis-pronouncing the name of the candidate, or getting the wrong voter on the phone. Or, maybe they are just forgetting to put one of the candidates in the poll. I have seen each of these errors in polling – and those errors can make a polling result fall well outside of the margin of error.

There has been a big concern about errors in polls given how hard it is to reach voters now that it’s not the 1950s and everyone isn’t just sitting at home next to their land line phone and willing to take a call from a stranger. Polling has had to evolve with the move to cellphones, and even branch out to obtaining completed survey via emails with links to online surveys and text messages with links to surveys that voters take on their phones.

Another key methodological difference in polling is who the pollster is reaching out to. Most commercial pollsters have the budget to pay for high-quality voter lists which will include all the voting behavior and other demographics to help them identify who to reach, before they start calling, texting and emailing. Polling from the voter file is the gold-standard as it allows the pollster to know the targets and know for sure that they are registered and if they are likely to vote based on their prior voting history.

There has been a big concern about errors in polls given how hard it is to reach voters now that it’s not the 1950s and everyone isn’t just sitting at home next to their land line phone and willing to take a call from a stranger.

In contrast to this best practice, some pollsters may go an entirely different direction, using a method called “Random Digit Dialing” which literally starts with the area codes and prefixes in a local area, and starts randomly calling people. In an RDD poll, the pollster may match the respondents to the voter file, but the more common approach is to just ask the voter if they are registered, and ask if they regularly vote in elections, in order to determine if they should be included in the poll.

Still another method is to use a “panel” which is a group of people who have agreed to take surveys – they could be political surveys or surveys about kitchen appliances – but these individuals will self-identify that they are voters in order to participate in surveys they are paid for. Of course, there are some variations on these three methods – like some panels could be created from registered voter lists, or some random digital dialing could be matched back to voter files.

The challenge for most political observers is that they hear about a poll in the news, but rarely is that polling result going to identify how the survey was done, how many people were polled and how it was weighted. They may have the margin of error – but not other information that speaks to the poll’s quality. It is almost like being sold a car, but you know nothing about it but the miles per gallon.

One thing that the release of so many polls could be encouraging is “herding,” the term for when pollsters are hesitant to release a poll that is outside the range of what others are showing. I personally know what this feels like. Earlier this year I had a poll we did in Los Angeles County where I saw District Attorney George Gascon losing by a 2:1 margin to his challenger Nathan Hochman. I kept quiet about it because it wasn’t a large sample, and I really didn’t want to put out some polling result that would look stupid. But, now that other polls are showing the same thing, I could be encouraged to release that same poll today. That’s herding.

There is also a kind of herding that can happen under the hood. Polls that do traditional weighting based on party/age/ethnicity may be encouraged to put more emphasis on income/education levels, simply because other pollsters are doing it. And, recently, we have seen a prevalence of “recalled presidential vote” in weighting. This works by asking a voter who they voted for in the last presidential election, then weighting the poll so there’s the right ratio of each candidate’s supporters, in line with the result from that are in the past election. Historically this has caused errors because voters may not answer this question correctly – either they lie, or they simply forget. If there is a big polling miss in 2024, I would expect you may hear a lot about weighting on recalled presidential vote.

Despite these points which probably sound like criticisms, public polling has generally been of relatively high quality. The 2022 polling was arguably the best that we have ever seen, at least since 1998 when land-line polling was the only game in town. And we are lucky to have so much public polling – literally hundreds of samples, some of which are from extremely reputable pollsters with great track records.

But the best quality polling is internal polling done by campaigns. These are the surveys done to tell a campaign or candidate how they are doing in a race, helping them test potential messaging, and guiding campaign strategy and spending. These polls can be very expensive, and it is extremely rare for a campaign to release an entire private poll.

Sometimes you will see a campaign release a polling summary in order to show candidate viability or drive a message to donors and political reporters. But when you see a release like that you should read it like it is a sales pitch, not the true poll. When a campaign is spending thousands of dollars on polling, the last thing they want to do is give all the real findings to their opponents for free.

The Polling Average
One of the most popular ways to analyze the presidential election is to avoid looking too closely at one individual poll, but instead look at the average of multiple public polls. And there are plenty of public polls out there to create an effective average that flattens out the highs and lows that appear in individual surveys.

Pollsters don’t create averages out of polls taken months apart, so this isn’t something that was commonly done in the business. Only in late stages of a campaign would internal pollsters do something similar using a “tracking poll” where they do small poll samples every day, and report to the client the three-day rolling average of those polls.

But the best quality polling is internal polling done by campaigns. These are the surveys done to tell a campaign or candidate how they are doing in a race, helping them test potential messaging, and guiding campaign strategy and spending. These polls can be very expensive, and it is extremely rare for a campaign to release an entire private poll.

But what multiple sites like Nate Silver’s creation FiveThirtyEight and his current project Silver Bulletin do is collect public polling, and put it into a rolling average. There are slight difference in polling averages because creators of the averages have to make some editorial decisions, like how long will a poll influence the average, and how much greater strength is given to newer polls. And some like FiveThirtyEight have a significant methodology to their averages. One basic technique is giving greater weight to polls by higher quality pollsters, so a poll by Ann Selzer will be given more weight than a poll by someone without a strong track record.

Additional they have started adjusting some polls for what is called a “house effect,” meaning that if a partisan pollster is consistently giving higher results for Republicans or Democrats, and that can be seen by looking at their historic polls against election results, they will dock that poll a couple points to balance it out. One recent gaming of the system we have seen is a large number of Republican pollsters “flooding the zone” with their polls, while few public releases of polls from Democratic firms. On one hand, this could mean that the polling averages are going to artificially show Trump performing better than he really is, although if averages are being adjusted for house effects, these polls aren’t going to impact the average as much.

Polling averages essentially build on the idea of a poll as a sample. But, instead of one poll of 400 or 1,000 voters, it allows for the aggregation of maybe 10 polls, including surveys from several thousand voters. There isn’t technically any margin of error applied when you combine different polls of different methodologies, but it is implied that these averages should provide a more accurate look at the electorate than just one poll.

Election Forecasts
Stepping past the polls and polling averages, we can get into the most recent and controversial method of analysis – the election model, and forecasting of potential outcomes. The most famous election model is the 538 projection which currently has Donald Trump winning 51 out of 100 times, effectively a 53% chance to 49% for Harris. Nate Silver’s new forecast has the race at a similar 54.7% for Trump to 44.9% for Harris.

In a way, the forecast is something most of us are extremely familiar with. Every morning I wake up and the first thing I check on my phone is the weather. How cold will it be today? Will I get rained on if I go out for a ride? How hot will it be this weekend?

The weather forecasts on our phones are based on multiple measures – they are partly using data on the fundamentals of the weather – it is hot in the summer, colder in the winter, then they use data from surrounding weather measuring stations which will pickup temperature and rainfall or wind, measures of barometric pressure, and data on nearby storms and changes in the Gulf Stream. All of this gets packaged on my phone as a simple temperature prediction throughout the day, and probability if there will be rain or wind, and how much.

Election forecasts work much the same way. They begin with Election Fundamentals – things like the power of incumbency, the burden of a bad economy, and other factors. They then blend into this data on the relative favorable ratings for the candidates and polling. Early in an election cycle the polling may not be as predictive, and so more emphasis is put on the fundamentals, but the closer we get to the election, the more the polls take center stage.

In a similar way, a weather forecast 10 days out is almost entirely the historic fundamentals – how hot is it usually on July 4^th at any given location – but at some point the weather forecast is going to put more weight on a nearby temperature sensor showing rain than the historic fundamental that it rarely rains on the 4^th.

There has already been one major nerd war over models after the disastrous debate performance by President Biden a month before he exited the race. At the time of the debate, most models showed the race in toss-up range. Trump had been leading in polls for months, but the economy was improving and Biden had the advantage of incumbency. However, after the debate, polls were shifting hard against President Biden. Nate Silver’s model correctly saw the impact – and his model dropped Biden’s chance of victory down to the 30% range. But the 538 model was still looking almost entirely at fundamentals, and still showing Biden with a roughly 50% chance of winning. The 538 model’s inability to capture the shifting voter sentiment was a real challenge, to the point where most began disregarding it.

While models have been put out by multiple outlets, causing many of us to addictively check them on a daily basis, the crack cocaine of all models is Nate Cohn’s dreaded Election Needle. Some may be having PTSD just reading this.

The Election Needle is an election-day specific model that kicks off projecting the election outcome based on the base models, using fundamentals and polling, then starts to mix in actual election results as they get reported. The needle looks like a fuel gage, with the Democratic candidate on one side, the Republican on the other. It will cause panic as it wildly vacillates to the right as one state posts its some results, then flips over to the left just as quickly to the other side as another state reports some county numbers. Before any major news outlet calls a race, campaign nerds are either celebrating or crying based on what all the needles are showing them.

Bottom line: watch the needle at your own risk.

Prediction Markets
If you’re too impatient to wait for a polling average to creep along, or for an election forecast to be run, you can always check the frenetic prediction markets. A prediction market is essentially a gambling site where people buy shares of “Trump wins” or “Harris wins” and the price of the share is effectively the predicted likelihood of each side winning.

The “wisdom of the crowd” theory behind prediction markets says that if a large, diverse and independent group of people hold an opinion, then that will usually be right. And you can see this in sports betting. With sports bets, if more people bet on a team to win, the betting line will adjust, and at the time of the game, the betting line will most often match the actual outcome. This is because those are receiving tons of bets, those bets are accessible to lots of people, and if those bets didn’t balance, casinos would lose, and they never lose.

Unlike sports bets where there is a betting line, in elections it works more like stocks, with value of any asset in a prediction market being priced somewhere between $0 – $1. If a market opens and a “share” of “Harris wins” is for sale at 50-cents, that means you can buy a share, and you will get $1 if she wins, $0 if she loses. If you think that is a good bet, you can buy shares – usually as many as you want. But, as you buy shares in Harris, the price will move up, and it will continue to rise until someone else who is more convinced that Trump will win sees that Harris shares are now 65-cents, and they can buy the other side of that, essentially buying “Trump wins” shares at 35 cents each. The price will see-saw back and forth as people who are convinced on either side bring the price to an equilibrium.

While models have been put out by multiple outlets, causing many of us to addictively check them on a daily basis, the crack cocaine of all models is Nate Cohn’s dreaded Election Needle. Some may be having PTSD just reading this.

In a perfectly efficient market with lots of players, a price of “Kamala Wins” at 33.3-cents, which it is at the time of this writing, means that everyone agrees that she has a 1-in-3 chance of winning, and Trump has a 2-in-3 chance. If “Kamala wins” believers actually thought that her chances were greater than 1-in-3 they would be leaving money on the table not betting with a chance of making so much money in a mis-priced market. As the theory goes, the lack of people buying Kamala shares at 33-cents is proof that this is her real likelihood of winning.

One thing that I have been critical about regarding betting markets as a prediction tool is that they really aren’t reflecting the wisdom of the whole. Remember, to be effective we want diverse and independent voices. But, instead, we often see that those betting are dominated by people with one partisan leaning or a subset of individuals who are neither diverse or independent.

As an example, Elon Musk has been pushing people to bet Trump, almost in an encouragement to bet in order to “own the libs.” Making matters worse, current federal laws restrict betting on these sites in the United States, so if there is betting happening here it is through setting up virtual private networks and using offshore money. These sites also take a lot of bets in Crypto, which is a method of payment with its own political leanings.

Additionally, there is a certain amount of friction in election betting. Getting money into one of these sites, paying transaction fees, hoping that you can actually get your money out when you want – all of these costs of participating in the market reduce the likelihood that normal people would even engage, even if there was an opportunity to take a bet they believe in.

This all goes back to the original Nate Silver critique of betting markets before he joined Polymarket an advisor. He and the 538 crew would refer to them as a “just a bunch of Scottish teens,” joking that these weren’t better informed crowds, but goofy kids, without any special information that was better than the polls.

In defense of betting markets, they are much nimbler than polls which take days to produce or models that may not be prepared to adjust for every event in real-time. As an example, at the moment of the first Trump assassination attempt, the betting markets instantly priced in higher odds for Trump winning, where it would take polls a long time to measure that impact, and no model created has a “in case of assassination attempt” calculation. Similarly, if in the next couple days there really is an October surprise or one of the candidates does something that fundamentally changes the race, the polls and models won’t catch it – but betting markets will.

It is hard to get around how if a race truly is a toss-up like the models and polling are saying that more money isn’t going in to buy Harris shares at just 33-cents. But if people are taking bets on the wrong side, at least they will be losing money as a punishment for being wrong – something that rarely happens in the world of election prediction and punditry. Betting markets enforce a kind of honesty, and a price for making bad predictions, which is a good thing.

Early Vote Tea Leaves
The early vote is well underway, with over five million ballots already being cast, and in-person voting centers open in most counties. The early vote is predictably older and whiter as early voters are more likely to be suburban homeowners who are the types who pay their bills by mail, and don’t leave things to the last minute.

The first question I usually get about the early vote is “how does this compare to 2020?” But that question makes me cringe, given that 2020 was such a crazy election with COVID lockdowns really impacting voting. Back then we were checking the mail every day, and things weren’t just sitting on the kitchen counter. Going out to drop off a ballot at a dropbox felt like a vacation! Then, of course, on the right Trump was pushing voters against vote by mail, and Democrats were pushing online fears about mail delays and encouraging their voters to return their ballots immediately.

As a result of how unique 2020 was, it has become a very unreliable baseline for any kind of analysis, as I wrote for the Sacramento Bee this week.

The early vote is fascinating to watch, and there can be lessons to learn from how total turnout is progressing, and what subgroups are returning their ballots early. But diving too deep into the data can be incredibly misleading.

But campaigns and the media continue to make comparisons, even where they aren’t warranted. One example on the national stage is how big early voting was in Georgia this year – a key swing state. It is true, the number of early votes skyrocketed, but it is wrong to suggest that this speaks to enthusiasm, or potential for higher total turnout. The fact is that in 2020 lines to vote in Georgia were over five hours in some places. Now, it can take you just 45 seconds to get checked in at an early voting center in Atlanta. So, comparing 2020 to 2024 as if you’re seeing a signal of enthusiasm is flawed – what you are seeing is mostly due to a change in the mechanics of the election.

The early vote is fascinating to watch, and there can be lessons to learn from how total turnout is progressing, and what subgroups are returning their ballots early. But diving too deep into the data can be incredibly misleading.

The 13 Keys
We cannot get out of an analysis of election prediction without getting into the absurdity that comes from American University professor Alan Lichtman who has for decades touted, and even written a series of books on the 13 Keys of the Election.

While he gets a lot of media attention for the confidence with which he uses history to predict the coming election, the fact is that his methods are akin to predicting the weather by how your dog acts in the morning, listening to birdsongs and measuring the wind. You might even get it right – maybe go on a string of getting it right for a couple weeks – but that doesn’t mean your system has any actual validity.

The fact is that there are so few presidential elections – only one every four years. Imagine if I came to you and said “I’ve got this poll that says who will win” and when you asked how many people were surveyed, I said oh, 46. You would laugh me out of the room. But there have only been 46 presidential elections, so building a kind of Keys of the Election prediction is based on too small a sample to be meaningful.

Professor Lichman is basically a hack selling election analysis snake oil. Unsubscribe.

Exit Polling
The final kind of polling doesn’t really fit into this article because it won’t do a thing to calm your nerves before the election. Exit polling is done by asking voters who they supported after they have cast ballots. For nearly a decade, Capitol Weekly has been doing these kind of exit polls of people who have returned mailed ballots – we have about 5,000 such completed surveys so far in 2024. But the value of these surveys is really to better understand the election after the fact.

Remember the discussion of weighting? Well, exit polls don’t predict election outcomes since those already happened, but they work the other direction – they weight the survey based on the actual result, and use that to try and identify the shares of the electorate that are from different subgroups – like how many young voters and Latinos and other subgroups voted.

It is noteworthy that the initial exit poll findings can be extremely wrong. In 2014 an exit poll pegged turnout of young voters at record highs, but we could see from the actual data of who had voted that turnout among young voters was assuredly low. Yet, article after article would cite the exit poll as if it was a more concrete finding than the actual data from county registrars showing who had voted. We see the same mistakes in exit polls that will sometimes show extremely high Latino turnout, but then when we run the actual data after the voter file are produced, find Latino turnout was below their rate of registration.

Exit polling can be interesting, but don’t use it to make sense of who voted in the election – simply wait for the voter files to be updated and you can have the actual data on who voted.