CA120: Crunching the poll numbers, big time

 This month has seen the release of dozens of new public polls, ranging from the presidential contest to statewide and local races.

We have seen many of these publicly available surveys, but the vast majority of polling is still private – done by candidates and political action committees. The results are rarely shared with those outside a very small circle of candidates and consultants.

With public polling, the consumer receives much more information than might have been available previously. But we have no input on the way the surveys are done, and little basis for understanding their potential flaws.

One great example of this is SurveyUSA, a national data firm known for inexpensive polling, often using a range of methods, like random-digit dialing and online surveys, that are not tied to actual voter records. This firm also has the annoying practice of weighting samples to match the adult voter population in California.

But that weighting does not take into account the fact that the registered voter population is vastly different.

For example, its latest survey weights results to a population that is 31% Latino in a state where the final electorate in a statewide election has never exceeded 20% Latino.  And when you skew a poll to be one-and-a-half times more Latino than it is supposed to be, you end up with results that are dramatically flawed.

Even armed with the same data, these pollsters all came to different results, ranging from a four-point lead for Hillary Clinton to a one-point lead for Donald Trump.

Yet, without an understanding of the methodology and potential flaws, reporters and political observers give the same weight to a public SurveyUSA announcement as a result from the Field Poll or Public Policy Institute of California.

This is like being told Folgers Crystals is a substitute for an espresso from your favorite independent coffee roaster, or that some guys in a half-court pickup game are just like the Golden State Warriors.

Nate Cohn of the New York Times has been dissecting these polls and providing information on how they are conducted. He has explored discrepancies between major national polls and shown how simple choices made by pollsters can change the outcome.

In one article, he obtained the raw polling data from a public poll in Florida, gave it to multiple pollsters and had them give him back the results.

Even armed with the same data, these pollsters all came to different results, ranging from a four-point lead for Hillary Clinton to a one-point lead for Donald Trump.  Each respected national pollster had the same data but came to different results based on how they weighted the data for ethnicities, partisanship, or their own models of likely turnout.

In a second article, Cohn dug in to the USC Daybreak poll, the only major national poll that has consistently shown Trump winning, and he exposed one of the most entertaining flaws ever found in public polling.

A single African American 19-year-old in Connecticut was being weighted so radically that his vote was counting as the equivalent of 30 responses.  And, unlike others in his demographic cohort, he is a Trump supporter, so when he was included in the poll, things got screwy. The USC survey was giving Trump a full-point advantage based on just one participant, and setting African American support for Trump as high as 12%, based almost entirely on this weighting flaw.

 Of the 945 respondents who answered the first measure, 933 completed through to the final.

Inspired by this work, we decided to run a polling experiment with Alan Yan, who previously worked on CA120 exit polling, and look at the 17 measures on the California ballot.

At first, our interest was to see if we could quantify the down-ballot impact of the extremely long ballot.  As was explored in an earlier CA120 article, there is significant evidence that voters begin to skip measures, or just vote no, as they work their way down the ballot.

This experiment grew as we began to write different versions of the survey. In the end, we were able to create three different surveys in order to explore more deeply how methodology impacts the results.

Voters deemed likely to vote received an invitation to take one of three surveys via email over a two-week period. The email addresses were publicly available from elections officials. These respondents were asked a range of questions before entering the battery of questions regarding the state’s ballot measures.  Once in the ballot measure sections, there were three versions:

• All measures listed in order on one single page. Voters could select Yes, No or Unsure.
• Selected measures presented in categories, regardless of position on the ballot. So, for example, the two plastic bag measures would be seen on one page, then the next page would have the two death penalty measures. Only 9 of the 17 measures were included.
• All measures listed on individual pages in the survey. When presented a question they could answer Yes, No or “need more information.”  When a voter said they need more information they were taken to a second page with the official ballot language Yes and No arguments, and then again given a chance to vote.

Version 1 – All Measures on One Page
This was the most straightforward way to present all 17 propositions and mimicked a poll-voting experience where one might be going quickly from one measure to the next.  Of the 945 respondents who answered the first measure, 933 completed through to the final.  The lowest response rate for any measure was Prop. 66 with just 928 completes, a very small drop-off.

The results of this poll showed numbers that were consistent with other public polling. Overall, most ballot measures appear to be in a strong position to pass, with the only real exceptions being the two death penalty ballot measures. Proposition 66, the measure to speed up the Death Penalty in California is only at 34% support with a lot of undecided, and Prop 62, the measure to eliminate the death penalty is at 40% No, a high level of opposition that has historically been hard for measures to overcome by Election Day.

Version 2 – Clustered
Providing fewer measures, and presenting them in a manner that allows for comparison, was intended to provide greater context for each measure.  It was understood that voters would not be seeing measures in this way on their ballot.  This method appeared to harm the Prop 55 tax extension and actually increased the percentage of undecided for each measure.  This methodology was also better at keeping respondents with the poll, and had very little drop-off from one section to the next.

 Version 3 – More Info version
The final version of the survey was the most elaborate.  Each measure was on its own page, and for each measure the respondent could ask for “more information” before they vote. This might be more like absentee voting, where someone can make a choice quickly, or look for more information in the sample ballot or online when they get stuck.

While the results of this method appear to have been more robust and informative, it also resulted in the greatest drop-off, a 15% total drop-off from beginning to end.  This, more than anything done in this set of surveys, supports the idea that voters might fatigue as they get through the ballot, and those at the end might suffer.

Yet, despite the drop-off, the responses to the initial answer, excluding those who asked for more information, matched the results from the other surveys. The average variance between this survey and the other two polls was just four points.

But, then, with the addition of the votes after getting more information, Proposition 59, the measure against Citizens United, gained 20-points, followed by Proposition 53 restricting Revenue bonds at 18-points and Prop 61, the Prescription Drug measure gaining 14.  Only the Tobacco tax actually lost “yes” percentage when voters got more information, while Props 60,62 and 67 each gained a few points in “no” vote.

With this additional information we saw our greatest variation in results among the three survey methods.  On average, the results of this combined base and follow-up question was 10-points among all measures.

The need for more information was itself interesting, showing which measures were potentially most confusing to voters, and which were well within the ability of voters to understand and make a quick judgement.

While most people have a clear position on the Death Penalty, the measure to extend capital punishment nearly half of the respondents asked for additional information.  At the other end of the spectrum, only one-in-twelve voters needed more information before voting on the tobacco tax.

Are voters confused?
With a much larger ballot this year, and a political climate very driven by the National political stage, not state issues, it could be expected that voters are headed to the polls with less information about the ballot measures. Additionally, there are at least three sets of ballot measures that are diametrically opposed to each other – these are the sets of measures on the Death Penalty, Plastic bags and, to a lesser extent, measures to increase or decrease taxation and bond revenue.

To explore this, we added a follow up question regarding sets of issues, asking if the voters viewed their votes as for or against an industry.  These can be juxtaposed against the actual votes on the measures themselves.

The findings of each of these questions is not by itself very informative.  California is definitely split on the death penalty, generally supportive of environmental measures like banning plastic bags, and against taxes in a general sense, although willing to support specific taxes for a defined need, like education or public safety.

Over half of voters who view their votes as anti-tax actually supported the Prop. 56 tax on tobacco,

Yet, a breakdown of the votes on each proposition, based on the results of these questions, shows a significant disconnect between how voters characterize their position and their actual votes.

• A whopping 68% of voters who supported Proposition 65, an initiative placed on the ballot by the plastic bag industry, incorrectly characterize their vote as against the industry. Another 12% of “No” votes on the plastic bag referendum come from people who think they are voting against the industry.
• Proposition 62, ending the death penalty, is supported by 92% of voters who claim that position, however, 40% of those same anti-death penalty voters are casting ballots for Proposition 66 which extends and speeds up the death penalty.
• Over half of voters who view their votes as anti-tax actually supported the Prop. 56 tax on tobacco, and smaller ranges of 30-40% supported Prop. 53 on Medi-Cal fees, Prop. 51 on School bonds and the Prop. 55 tax extension.  A majority of voters who describe their votes as anti-tax did support Prop. 53 which restricts use of revenue bonds and could be characterized as an anti-tax measure.

Remaining questions
This experiment of using three different methods to analyze their impact on polling results provided some examples of how decisions made by pollsters and campaigns might impact the final read on a race.  The most dramatic difference was the use of a “more information” option and a second opportunity for a voter do make a selection.

Two elements that this polling method did not give us clear answers on are the likely voter drop-off and the potential that voters might vote on the first several issues, then start getting fed up and voting no on everything else.

We did see drop-off from the version of the survey that required the respondent to click 17 times to get through the entire questionnaire. That might mimic the frustration of having to vote on all 17 measures, but we did not see any gravitational pull toward “no” as the ballot wore on.

As a part of understanding more about the drop-off, our statewide absentee voter exit poll (which is currently in the field and currently has over 4,000 responses) asks voters how they vote in a long ballot with limited information.  So far, 30% are agreeing with the statement “I find the races that I care about, and focus on voting on those, leaving some blank.”

The poll of the polls*
What many really want to know is how, after all these methods are combined, the ballot measures are doing.  One very simple method is to just combine all the votes in each of the surveys and come up with a final set of numbers.

Below is a table with each measure and their result among the combined 2,637 respondents.  For the third version of the poll with the “more information” option, we included just the initial response, so this underrepresents the likely total support for all measures except for Prop 56, the tobacco tax. Overall, this shows very strong performances for nearly all of the ballot measures and the likelihood that a dozen or more of them are headed to victory based on this and other public polling.

*Ed’s Note: CORRECTS transposed numbers in Poll of Polls graphic, UPDATES earlier.
—
Ed’s Note: Paul Mitchell, a regular contributor to Capitol Weekly, is the founder of the CA120 column and the vice president of Political Data, Inc., which markets campaign information to campaigns in both major parties. Alan Nigel Yan, a former UC Berkeley intern, conducted these polls with data and tools from Political Data, Inc., from Sept. 25 to Oct. 15.