Jon Krosnick on elections and polling
October 30, 2008
The human mind being an inexplicable sort of thing, we aren’t able to ask people simple questions like “Why did you vote the way you did?” and expect an accurate answer. We don’t know squat about memory or decision making or any of those things that would influence a vote, so the best we can do is to simulate the voting situation with polls. The most historically accurate polls on presidential elections correlate 95% with the actual vote – but 2008 results won’t be out until 2009, making this an academic rather than predictive tool.
A certain 1990 race for attorney general in Ohio ended in a narrow defeat for one candidate, Pfeifer, and a subsequent lawsuit. Among Pfeifer’s complaints: his name appeared second on the ballot, and voters were more likely to vote for the candidate whose name appears first.
In fact, Ohio state law requires that the order of candidates be rotated on the ballot, for exactly this reason. Half the precincts should have had Pfeifer’s name first, but only about 43% did. They brought in experts to determine whether the mistake skewed the vote, and Krosnick’s group compared the results from precincts that put Pfeifer’s name first versus his opponent’s.
They did find a primacy effect (that is, the first name gets more votes), to the tune of a fraction of a percent. That wasn’t enough to have cost Pfeifer the election, so the case was closed.
Except that to find a small primacy effect was surprising – most elections have a large primacy effect. In fact, when people are asked to choose their favorite ice cream in a taste test, or to guess on multiple-choice quiz questions, or just to pick random words off a sheet of paper, the choices earlier in the list always get more than their fair share of attention.
In Ohio’s 1992 elections, the researchers found a 3% primacy effect in many of the races. That’s 3% each way, making a 6% split on average (the largest effect was 6.27%). Various studies agree that there is always a primacy effect on ballots, although the studies disagree about just how big that effect is. Even the smaller results, like a 0.14% effect in certain Ohio elections, are still enough to influence an election. Gore lost to Bush in Florida in 2000 by a margin far smaller than that.
Why would there be a primacy effect in elections? There isn’t enough information to know, but it’s likely that some undecided voters feel that they have to pick a candidate, and pick one at “random”.
They may be ambivalent about their choice; primacy effects are stronger in more obscure races, and smaller when there is a lot of media coverage. The effect is also strong when there is no incumbent candidate. (Krosnick thinks that’s because people are judging the incumbent’s qualifications by his record; I suspect people will just vote for the name that seems most familiar.) More evidence that voters are guessing: the effect is strong when the candidates’ party isn’t listed with their name.
A study of big-ticket California races from 1976-2000 showed significant primacy effects in 85% of the races. California is another state where candidates’ names must be rotated. That means that in precincts where the democrat was first on the ballot, the democrat got more votes; but in the next precinct over, the republican would be first on the ballot and cancel out the advantage. (It is possible, however, for the effect to be stronger for one candidate than the other … I don’t have a good explanation for that.)
States vary widely in their laws on rotating names on ballots. Some states require it, and some forbid it. Some mandate that the democrat is always first (but none require the republican to be first). In case you were wondering, Bush’s name came first on every ballot in Florida in 2000. The margin of his win was much smaller than the primacy effect he likely benefited from. “We heard a lot about butterfly ballots, but I think this is the real scandal of Florida in 2000,” he said.
(Fun fact: in 2001, eleven news organizations chipped in to hire a team of consultants to do a full manual recount of the Florida votes. It took a long time and a lot of money, but they reported that if such a recount had been allowed in November, Gore would have come out ahead.)
In Mahoning County, Ohio, touchscreen voting machines had replaced paper ballots in a certain election, so that when the researchers asked about the order of names, the board of elections couldn’t say. The computer was programmed to swap the names on a per-voter basis, and there is no way to know which voters got which order. The county seemed pretty proud of themselves for eliminating a source of bias, Krosnick said, until he pointed out that there is no way to monitor the bias, or lack of it. “And by the way, do you know you’re violating state law?” he asked them. Remember, Ohio requires rotating by precinct.
Interestingly, pollsters often remember to rotate the names on their polls, but that can lead to inaccurate results in states that don’t rotate. Krosnick thinks this happened in the 2008 primary in New Hampshire. The surveys all rotated the names when they asked people who they planned to vote for, but when voters got to the voting booth, Hillary Clinton was first on the ballot. And, contrary to the polls, she ended up with the greatest number of votes in that state.
There is another implication for telephone polling: although a primacy effect reigns among written ballots, oral polling has the reverse effect: the name you heard last is the one you’re likely to latch onto.
Interestingly, polls always list the candidates. Nobody does polls where they just say “Who are you gonna vote for?” and wait for an answer. Those would be interesting, but since they don’t mirror the ballot, they’re less likely to be accurate.
Skipping to the question of exit poll accuracy, Krosnick points out that they tend to have a democrat bias, and democrats are always listed before republicans on the exit polls (because D comes before R, I guess). That may be why.
The predictors of exit poll accuracy are the location of the interviewer relative to the doors of the polling place; having a small number of precincts at the same polling place (these may have to do with the interviewer obtaining an accurate sample) and the number of respondents – but not the response rate.
In pre-election polls (which Krosnick points out are getting more accurate every year), the ideal poll would include a random sample of the country, randomly selected households, randomly selected voters within those households (NOT just the first person to pick up the phone), and the largest samples possible – which rules out single-day polls. He says the response rate is not as important as the number of people asked, since the first few people to respond tend to be representative of the total pool of respondents. (This doesn’t quite make sense to me, but that’s what he said.) The ideal poll also doesn’t ask warm-up questions since those tend to bias people’s answers; and importantly, the ideal study weights the respondent pool to mirror the demographics of likely voters. Which assumes that you know who the likely voters are, and that’s a difficult question.
The oft-cited “Bradley Effect” is an example of a pre-election poll whose results were way off. The theory goes that voters told interviewers they were likely to vote for Bradley, who was black, but in the privacy of the voting booth they voted against him. That election became a famous example of implicit racism. But studies of more recent elections fail to show a Bradley Effect. There is a known race-of-interviewer effect (where you’d be more likely to tell a black interviewer you’re voting for the black candidate) but surprisingly, this year’s polls have shown no such effect.
Krosnick’s group did a study looking for effects of racism in the choice between McCain and Obama. They found an effect of explicit racism – that is, voters who admitted to being racists did in fact favor McCain – but there was no additional effect of implicit racism. They attribute this to a fairly detailed survey, saying that if you just boil it down to “Are you a racist?” not many people will say yes.
The researchers tried to figure out what factors were important to people in their choice between the two candidates. Racism, we’ve seen, favors McCain. (There were voters who favored Obama specifically because of his race, but far fewer than the reverse.) Many of the factors they looked at showed no predictive value, but here are the ones that did:
• Identify as a republican – favors McCain
• Competence – favors McCain
• Military experience – favors McCain
• Racism – favors McCain
• Familiarity – favors Obama
• Ability to bring change – favors Obama
• Cares about my issues – favors Obama
• Opinion of President Bush – favors Obama
• Opinon of Michelle Obama – favors Obama
• Opinion of Cindy McCain – favors McCain
The study was done in August, so obviously a lot has changed since then. But these results show an interesting snapshot.