This is the sixth in a series of posts about the recent Science Writers conference. Here, my notes from Jon Krosnick’s talk about elections and polling.

The human mind being an inexplicable sort of thing, we aren’t able to ask people simple questions like “Why did you vote the way you did?” and expect an accurate answer. We don’t know squat about memory or decision making or any of those things that would influence a vote, so the best we can do is to simulate the voting situation with polls. The most historically accurate polls on presidential elections correlate 95% with the actual vote – but 2008 results won’t be out until 2009, making this an academic rather than predictive tool.

A certain 1990 race for attorney general in Ohio ended in a narrow defeat for one candidate, Pfeifer, and a subsequent lawsuit. Among Pfeifer’s complaints: his name appeared second on the ballot, and voters were more likely to vote for the candidate whose name appears first.

In fact, Ohio state law requires that the order of candidates be rotated on the ballot, for exactly this reason. Half the precincts should have had Pfeifer’s name first, but only about 43% did. They brought in experts to determine whether the mistake skewed the vote, and Krosnick’s group compared the results from precincts that put Pfeifer’s name first versus his opponent’s.

They did find a primacy effect (that is, the first name gets more votes), to the tune of a fraction of a percent. That wasn’t enough to have cost Pfeifer the election, so the case was closed.

Except that to find a small primacy effect was surprising – most elections have a large primacy effect. In fact, when people are asked to choose their favorite ice cream in a taste test, or to guess on multiple-choice quiz questions, or just to pick random words off a sheet of paper, the choices earlier in the list always get more than their fair share of attention.

In Ohio’s 1992 elections, the researchers found a 3% primacy effect in many of the races. That’s 3% each way, making a 6% split on average (the largest effect was 6.27%). Various studies agree that there is always a primacy effect on ballots, although the studies disagree about just how big that effect is. Even the smaller results, like a 0.14% effect in certain Ohio elections, are still enough to influence an election. Gore lost to Bush in Florida in 2000 by a margin far smaller than that.

Why would there be a primacy effect in elections? There isn’t enough information to know, but it’s likely that some undecided voters feel that they have to pick a candidate, and pick one at “random”.

They may be ambivalent about their choice; primacy effects are stronger in more obscure races, and smaller when there is a lot of media coverage. The effect is also strong when there is no incumbent candidate. (Krosnick thinks that’s because people are judging the incumbent’s qualifications by his record; I suspect people will just vote for the name that seems most familiar.) More evidence that voters are guessing: the effect is strong when the candidates’ party isn’t listed with their name.

A study of big-ticket California races from 1976-2000 showed significant primacy effects in 85% of the races. California is another state where candidates’ names must be rotated. That means that in precincts where the democrat was first on the ballot, the democrat got more votes; but in the next precinct over, the republican would be first on the ballot and cancel out the advantage. (It is possible, however, for the effect to be stronger for one candidate than the other … I don’t have a good explanation for that.)

States vary widely in their laws on rotating names on ballots. Some states require it, and some forbid it. Some mandate that the democrat is always first (but none require the republican to be first). In case you were wondering, Bush’s name came first on every ballot in Florida in 2000. The margin of his win was much smaller than the primacy effect he likely benefited from. “We heard a lot about butterfly ballots, but I think this is the real scandal of Florida in 2000,” he said.

(Fun fact: in 2001, eleven news organizations chipped in to hire a team of consultants to do a full manual recount of the Florida votes. It took a long time and a lot of money, but they reported that if such a recount had been allowed in November, Gore would have come out ahead.)

In Mahoning County, Ohio, touchscreen voting machines had replaced paper ballots in a certain election, so that when the researchers asked about the order of names, the board of elections couldn’t say. The computer was programmed to swap the names on a per-voter basis, and there is no way to know which voters got which order. The county seemed pretty proud of themselves for eliminating a source of bias, Krosnick said, until he pointed out that there is no way to monitor the bias, or lack of it. “And by the way, do you know you’re violating state law?” he asked them. Remember, Ohio requires rotating by precinct.

Interestingly, pollsters often remember to rotate the names on their polls, but that can lead to inaccurate results in states that don’t rotate. Krosnick thinks this happened in the 2008 primary in New Hampshire. The surveys all rotated the names when they asked people who they planned to vote for, but when voters got to the voting booth, Hillary Clinton was first on the ballot. And, contrary to the polls, she ended up with the greatest number of votes in that state.

There is another implication for telephone polling: although a primacy effect reigns among written ballots, oral polling has the reverse effect: the name you heard last is the one you’re likely to latch onto.

Interestingly, polls always list the candidates. Nobody does polls where they just say “Who are you gonna vote for?” and wait for an answer. Those would be interesting, but since they don’t mirror the ballot, they’re less likely to be accurate.

Skipping to the question of exit poll accuracy, Krosnick points out that they tend to have a democrat bias, and democrats are always listed before republicans on the exit polls (because D comes before R, I guess). That may be why.

The predictors of exit poll accuracy are the location of the interviewer relative to the doors of the polling place; having a small number of precincts at the same polling place (these may have to do with the interviewer obtaining an accurate sample) and the number of respondents – but not the response rate.

In pre-election polls (which Krosnick points out are getting more accurate every year), the ideal poll would include a random sample of the country, randomly selected households, randomly selected voters within those households (NOT just the first person to pick up the phone), and the largest samples possible – which rules out single-day polls. He says the response rate is not as important as the number of people asked, since the first few people to respond tend to be representative of the total pool of respondents. (This doesn’t quite make sense to me, but that’s what he said.) The ideal poll also doesn’t ask warm-up questions since those tend to bias people’s answers; and importantly, the ideal study weights the respondent pool to mirror the demographics of likely voters. Which assumes that you know who the likely voters are, and that’s a difficult question.

The oft-cited “Bradley Effect” is an example of a pre-election poll whose results were way off. The theory goes that voters told interviewers they were likely to vote for Bradley, who was black, but in the privacy of the voting booth they voted against him. That election became a famous example of implicit racism. But studies of more recent elections fail to show a Bradley Effect. There is a known race-of-interviewer effect (where you’d be more likely to tell a black interviewer you’re voting for the black candidate) but surprisingly, this year’s polls have shown no such effect.

Krosnick’s group did a study looking for effects of racism in the choice between McCain and Obama. They found an effect of explicit racism – that is, voters who admitted to being racists did in fact favor McCain – but there was no additional effect of implicit racism. They attribute this to a fairly detailed survey, saying that if you just boil it down to “Are you a racist?” not many people will say yes.

The researchers tried to figure out what factors were important to people in their choice between the two candidates. Racism, we’ve seen, favors McCain. (There were voters who favored Obama specifically because of his race, but far fewer than the reverse.) Many of the factors they looked at showed no predictive value, but here are the ones that did:

• Identify as a republican – favors McCain
• Competence – favors McCain
• Military experience – favors McCain
• Racism – favors McCain
• Familiarity – favors Obama
• Ability to bring change – favors Obama
• Cares about my issues – favors Obama
• Opinion of President Bush – favors Obama
• Opinon of Michelle Obama – favors Obama
• Opinion of Cindy McCain – favors McCain

The study was done in August, so obviously a lot has changed since then. But these results show an interesting snapshot.

This is fifth in a series of notes from the ScienceWriters2008 meeting. Lisa Bero gave a talk on “Faulty clinical trials and financial conflicts of interest.” [10:18]

Lisa Bero started off with a diagram of “the cycle of bias in research”. Your research question influences the population you test it on, which influences your method, which influences how you conduct the study, which influences what you publish.

A published study influences the questions that form in other scientists’ minds as they set off on their own cycles, but can also affect meta-analyses based on the research, and can filter into guidelines for patient care.

One of the sources of bias Bero is most concerned about is, simply, financial conflicts of interest. In 2004, the National Cholesterol Education Program updated its guidelines for cholesterol treatment, and overnight the number of Americans who should be on cholesterol-lowering meds shot from 13 million to 40 million. Eight of the nine experts recommending the change had personal financial ties to drug companies who make statins, and the evidence they relied on came from five randomized clinical trials (RCT’s) that were all funded by makers of statins.

Why worry about the funding of studies, though, if the study has been reviewed and the science is good? It turns out, in one of Bero’s own meta-analyses, that a drug company funded study is four times more likely to turn up a result favorable to its own product than is an independent study.

While clinical trials tend to compare a new drug to a placebo, a better way to study bias is in head-to-head comparisons that pit two competing drugs against each other. If Drug A fares better than Drug B, you would expect the same results no matter who funds the study. But it turns out that Company A’s results are the reverse of Company B’s for the same comparison.

There are many possible reasons for this bias, which Bero couldn’t pin down for sure. The difference could be in how scientists frame their question, how they design the study, how they conduct the study, and in the decision of whether a given result is worth publishing.

One source of bias that is fairly easy to demonstrate is dosage. If you want to compare your drug’s effectiveness to your competitor’s, you can give your drug in a higher dose and Brand X at a much lower level. If the drugs are equivalent, patients will experience more relief from the higher dose. On the other hand, if you want to show that your competitor’s drug has more side effects, you’ll give that drug in a higher dose than your own. The trick is to test those hypotheses separately, which is often the case.

In an analysis of 56 NSAID trials, 40 showed similar results from both drugs, 16 showed a stronger effect from the manufacturer’s own drug, and zero came out in favor of the competitor. Bero’s team compared these results to the dosage of the drugs (in terms of their own dose-response curves, of course) and found that the competitor’s drug was usually given at a very low dosage – no wonder it had less of an effect.

Another sneaky way of reporting results in an extra-flattering light is to focus on the p-value (if you have a good p-value to show). This is the number that determines “statistical significance”, and statisticians like to see it at a level of 0.05 or less.

In one study about the constipation drug Zelnorm, the p-value showing its effectiveness was p < 0.0001. That means the researchers are 99.99% sure that Zelnorm caused patients to poop successfully – sounds great, and as far as we know the p-value was correct.

The problem (aside from the fact that it caused heart attacks, which is a different story) is that it only had that effect in fairly few patients. With a placebo, 27% of patients had one extra poop per week. With Zelnorm, 40% did. There are several ways you can report that difference numerically, but one of the most intuitive (according to Bero, among others) is the NNT, or Number Needed to Treat. In this case, you have to give Zelnorm to 7.4 people to get just one patient with the desired (tee-hee) outcome. (Here is some more discussion of NNT – be sure to check out the chart on the side.)

So with those numbers, the drug didn’t work very well, even though the result was statistically significant. It caused the desired pooping, but not in very many patients. If you took Zelnorm, you had less than a 1 in 7 chance of experiencing that statistically significant effect. Bero ran some numbers up on the screen – it worked out to $155 per poop.

Boro then looked at a well-studied drug whose name I don’t remember. In the first study, the desired effect was observed, but the confidence interval was so large as to be meaningless. When you add in participants from the next study, and the next, and the next, the confidence interval shrinks substantially. That’s a good thing – it means that the drug’s effect is being pinned down to a narrow range of numbers. After about 5000 patients’ worth of studies, it’s obvious what the drug’s effect is, and that the confidence interval puts it squarely in the “definitely better than a placebo” category. And yet dozens more studies were done after that, showing the same effect of the therapy.

Bero says many of the studies were small, and done as marketing studies. A doctor would be asked to put 1-3 patients into the study, and the company would send out drugs for those patients. The point of the trial was to get the drug out to the community and get people talking about it, not primarily to test its effects. And so Bero says we would be better served by a few large, well-executed trials than by dozens of dinky marketing studies. (But then what would that do to her claim to “don’t ever believe a single study, even if I do it”? She’s a big fan of meta-analyses, which have their own problems.)

When a company develops a drug, they begin by working out its pharmacodynamics, pharmacokinetics, and do animal studies. Clinical trials come in three phases:

  • Phase I is done on healthy people, to gauge its safety
  • Phase II consists of small studies on sick people
  • Phase III trials are very large studies on sick people.

After that, a New Drug Application is submitted to the FDA, where various personages make reports and recommend risk management plans – and in about 20% of cases, a review board looks at the application. Surprisingly, most drugs don’t get that scrutiny. Then material from the NDAs can be made available, although Bero and her colleagues have found that large sections are redacted – including the researchers’ conflicts of interest, and sometimes important data like inclusion criteria and risk management recommendations. The redacted material is supposedly information that will harm the company’s competitive advantage, but (according to Bero, and really, to common sense) that information needs to be made available.

And soon, much of it will be. Companies submitting NDAs are now required to register their trials at ClinicalTrials.gov if they want to be able to publish the results. A new law, Public Law 110-85, states that all “basic results” must be included, although adverse event reporting is still optional. “Basic results” include participant flow (how many people started, completed, dropped out, and were excluded from the trial), the characteristics of the population, values of outcomes, and a contact person who can answer questions about the study.

Bero’s group will soon publish a study (I’m on the edge of my seat) comparing the information in NDAs to the published accounts of the same trials. We got a sneak peek of that data, which looks very interesting.

Now that we know about bias in clinical trials, what can we do about it? The options include banning conflicts of interest (which is what the Cochrane Collaboration does), putting restrictions on what the companies can do (such as requiring them to report data to ClinicalTrials.gov), and disclosure of the conflicts. Disclosure is the current situation, and it’s difficult to enforce. Many studies don’t state conflicts of interest, and journals usually don’t require such statements. Disclosure doesn’t prevent bias, Bero says, and actually might make it worse – since researchers figure readers will be taking their results with a grain of salt.

In Italy, drug companies have to pay a certain amount of money to the Italian version of the FDA when they submit their applications. This money goes to fund areas “where commercial research is insufficient,” namely, orphan drugs; head-to-head comparisons; and safety studies. This approach might not be practical in our government structure (it helps that Italy has a national health service) but it sure sounds like a good idea.

Upon request, Bero gave some advice for patient advocacy, suggesting Consumers United for Evidence-based health care (CUE) as a resource. Key areas of concern are conflicts of interest on FDA advisory committees, and the openness of NDA data.

For writers, the critical questions to ask include

  • Why was the research done – is this just a marketing study?
  • Who controls the research?
  • How did it get published?
  • Transparency of the methods
  • Funding, and researchers’ personal conflicts
  • Are there unpublished data? Sometimes data points are excluded for interesting reasons.

When reading a study, Bero suggests two key places to look for evidence of bias. First, what subset of the results makes it to the paper’s conclusion (and abstract)? Is anything important missing? And secondly, she says it’s always worth doing the math of how many subjects were enrolled in the study, and on how many are still there at the data reporting stage?

Another interesting thing to look at is the results – are the researchers measuring an actual outcome, like a number of heart attacks or deaths, or are they using a “surrogate outcome” like the results of a lab test? Drugs can be approved on the basis of just those lab tests. Surrogate outcomes are the basis for much of the recommendations for cholesterol-lowering statin drugs, for example, but those drugs may not have real-world effects with any kind of reasonable NNT.

A cat and mouse game

October 28, 2008

cat and mouse

cat and mouse

This is fourth in a series of posts about the Science Writers 2008 conference. Here are my notes from Robert Sapolsky’s plenary lecture on “Stress, parasites, and human behavior.”

Haven’t we all heard of toxoplasmosis by now? (Well, I guess it hasn’t gotten any less interesting.)

When a parasite invades its host, its job is not over – it has to create the conditions in which it can reproduce. If a host doesn’t want you there, what can you do?

HIV avoids becoming a victim of the immune system by attacking that system first. Since the immune system targets antigens by their surface proteins, one organism [didn’t catch which one] takes on the host’s surface proteins as a disguise. Since it takes about 14 days to mount an attack on an intruder, trypanosomes change their surface proteins every 13 days or so.

crab digging ... but for whose eggs?

Some parasites change their host’s behavior, including parasitic barnacles that prey on crabs. The barnacle somehow makes the crab more attractive to mates, and lays its eggs on the lover crab, when such a crab comes along. There is a mite that rides on an ant’s head, and knows how to stroke the ant’s mandible to make it regurgitate food. The mite is essentially fooling the ant into thinking it’s feeding its larvae.

Yet another barnacle parasite “feminizes” male crabs, making its host dig as if for a nest. But instead of laying crab eggs in the hole, the barnacle lays its own eggs there. This parasite also targets female crabs, destroying their gonads before triggering the digging behavior.

Rabies is a famous brain manipulator. Its particles are shed in saliva, so it changes the canine host’s behavior to make it more likely to bite other potential hosts. Neuroscientists are still struggling to understand the pathways in the brain that influence aggression – but a little virus called rabies has this system all figured out.

There is a bacterium that can only reproduce sexually in a cow’s digestive system. So it gets pooped out, eaten by an ant, and manipulates that ant into climbing to the top of a blade of grass. When it gets out of the lawn’s understory, the bacterium – which is photosensitive – paralyzes the ant’s mandible. The idea is that it freezes just as it’s biting into the blade of grass, and finds itself stuck there. Then a cow comes along, and eats it.

Toxoplasmosis has a similar problem – it only reproduces in the feline gut. Cat poop gets eaten by rodents, and the toxo has to find a way of getting its rodent host into a cat’s stomach.

So it seems the rodents seek out cats, rather than running in fear from them. When I first heard that idea, I thought it was unlikely – that the rodents probably are just careless around cats, or maybe they just can’t run away as effectively when they’re sick. The scientists thought of this too, and there is now a lot of experimental evidence supporting the idea that rats very specifically seek out the smell of cats.

Rodents normally avoid cat scent. Even born-and-bred lab animals that have never met a cat in their lives show an innate aversion to the scent. Except, that is, if they’re infected with toxo. There is even a dose-response curve like you would see with a drug.

And meanwhile, everything else about the rat is normal. Its sense of smell, its social behaviors, its ability to learn, and even its ability to learn to fear things. Its anxiety about other stimuli (everything dangerous except cats) is completely normal. The effect extends to the smell of both domestic cats and bobcats, but not to the scent of other animals they’ve tested, including dogs and humans.

By attaching a fluorescent tag to the toxo organism (which, by the way, is a protozoan), you can watch where in the rodent it localizes. It spreads throughout the body, causes various symptoms, and disappears from most areas of the body – except the brain.

Where in the brain? It seems it especially goes to the amygdala, which is known to have a major role in fear and anxiety. There, it causes dendrites to atrophy, essentially disconnecting the neurons from each other.

Here my notes are a little fuzzy, but the upshot is that, when a normal rat smells cat urine, fear-related areas light up. When a toxo-infected rat does the same, sex areas of the brain light up. Toxo makes cat urine smell sexy to male rats.

Don’t ask me why they’ve only tested male rats so far. Sapolsky doesn’t have data yet for female rats, but he thinks he will learn that, for toxo-infected females, cat urine smells like babies. We’ll see about that…

Blasting bits of toxo’s genome, you get hits for two of the enzymes involved in making the neurotransmitter dopamine. This seems to be unique to toxo; related protozoa, like plasmodium, have no such gene.

What’s the exact role of dopamine here? I’m not sure. This talk left me with many more questions than answers. Apparently, only a very small number of papers have been published on the mysterious behavioral effects of toxo. In humans, toxo is only well-studied in its effects on fetuses, which can include retardation, seizures, blindness, and death. If the kid survives, though, its post-toxo life isn’t studied. But toxo may have long-term behavioral effects in humans as well as in rats.

In a psych battery, toxo patients show slight differences from uninfected people in impulse control – for example, in a test where you have to switch from counting backwards to counting forwards, they have trouble making that switch on command. The effects are subtle in men, and even more subtle in women. But on a less subtle note, people infected with toxo are two to four times more likely to die in a high-speed car crash.

A popular hypothesis now is that toxo, as Sapolsky put it, “makes us enjoy making our body hurtle through space uncontrollably.” Anecdotally, transplant surgeons see toxo more often in organ donors who died in motorcycle accidents. The theory goes that toxo makes risky things seem somehow attractive. And toxo is very common in humans – 70% incidence in some tropical areas, and possibly 30% worldwide. There may be a connection between toxo and schizophrenia or psychopathy.

Sapolsky closed by saying the lesson here is one of phylogenetic humility – rabies and toxoplasmosis know more about how the brain works than our best neurologists.

wood rat homes

wood rat homes

Third in a series of things I learned today.

This is a wood rat midden that I saw at Jasper Ridge.

The grandma rat owns the house, and her daughters live in various parts of it. There are accessory houses nearby with walkways connecting them.

Sometimes the researchers need to work where the nests are, so they will either move the whole nest, making sure to keep a walkway connecting it, or at least leave the sticks around for the rats to reconstruct it. The rats can reconstruct them easily; apparently the bottleneck for them is actually finding the twigs.

Some of these houses are 30 years old.

I am not making this up. Second in a series.

Ticks that bite western fence lizards are cured of Lyme disease.

Serpentine grasslands

October 28, 2008

This iguana is made of the mineral Serpentine

This iguana is made of the mineral Serpentine, although Lizardite would be more appropriate.

This is the first in a series on interesting things I learned this weekend.

Serpentine is a mineral that comes from deep within the earth, around the mantle layer. At fault lines, it can come to the surface. California has many such fault lines, and in fact Serpentinite is California’s state rock.

Where this mineral comes to the surface, like at Stanford’s Jasper Ridge Biological Preserve, you’ll find toxic serpentine soil. It’s full of metals like nickel, cobalt, and magnesium; and it doesn’t have enough of basic nutrients like nitrogen, potassium, or phosphorus for most plants to live.

And yet there are plants that live in it, like tarweed, which our gude said she knows as “the smell of California in the summer”. They grow well enough in normal soil, but these species have the toxic serpentine all to themselves.

Aspirin in the air

October 7, 2008

Aspirin, by Chaval Brasil

Aspirin, by Chaval Brasil.

Scientists testing the atmosphere above a walnut grove in California found aspirin in the air.

Actually, they found a chemical variant of aspirin, methyl salicylate (MeSA), which you may not have heard of but you’ve certainly smelled and tasted. It’s also known as oil of wintergreen, ingredient of Ben-Gay liniment and wintergreen Life Savers. (It’s both the flavoring in those Life Savers and the reason they sparkle when you chew them).

Scientists from the National Center for Atmospheric Research say that plants use MeSA as a warning chemical, puffing it into the air to communicate “ecosystem-level stresses” to other plants.

Researchers had found earlier that tobacco plants release MeSA into the air when they are infected with a virus. The NCAR scientists found it above the walnut grove when temperatures got cold enough to damage leaves. The signal may give undamaged plants a head start on beefing up their defenses.

Aspirin-like chemicals seem to trigger a plant’s Systemic Acquired Resistance response, which is essentially the plant version of an immune system. The SAR response helps plants both to resist disease and to recover from it.

photo by joebaz

Willow bark. Photo by joebaz.

Aspirin is better known, of course, for its effects on disease symptoms in people. Its variant salicylic acid was known as a pain reliever and fever reducer long before it was chemically formulated in 1853 (as acetylsalicylic acid) or manufactured in pill form (by Bayer in the 1890s).

The bark of willow trees contains a lot of salicylic acid, and was used by Native Americans and ancient Sumerians, to name a few. The flowering shrub called myrtle and, of course, wintergreen, are also sources of pain relief preparations.

But with this new information about how plants use MeSA, people can now make use of the chemical in another way. Farmers could use the MeSA signal much like plants do – to detect stress or disease before it spreads. Since the MeSA signal appears before there is any visible damage to the plants, a farmer who monitors the signal can take action – applying pesticides, for example. “The earlier you detect that something’s going on, the more you can benefit in terms of using fewer pesticides and managing crops better,” is how one of the scientists put it.