Experiments Natural and Experiments Proper

10. Experiments Natural and Experiments Proper

Since newborns have immature immune systems, every effort should be made to minimize their contact with bacteria and viruses that cause diseases. —From Germ Fighting Tips for a Healthy Baby, CNN TV News, February 2, 2011 (CNN, 2011)

Infants who come into contact with a wide range of bacteria very early in life appear to be at a lower risk of developing allergies later in life … —From Infants’ Exposure to Germs Linked to Lower Allergy Risk, Canadian TV News, November 3, 2011 (CTV, 2011)

Friends, colleagues, and the media bombard us with advice about how to live our lives and conduct our professional activities. At the time I wrote this chapter, changing advice about dietary fat, vitamin B6 supplements, and red wine offered examples of a familiar problem: reports about the same topic could appear to point in different directions.

Even if we’re willing to default to accepting whatever is the latest medical advice, we have to reckon with conflicts between claims. Cousin Jennifer’s dentist recommends twice-daily flossing and your own dentist advises that occasional flossing is adequate. The New York Times’s financial advice writer tells you to dump stocks and buy bonds. The Wall Street Journal’s guest columnist’s advice is to buy real estate and park a lot of money in cash. Your financial advisor recommends a heavy allocation to commodities. Your friend Jake’s financial advisor urges shifting funds from domestic to foreign stocks.

Your friends Eloise and Max are extremely anxious about getting their child into the best possible pre-K child care at whatever cost; your friends Earl and Mike think that the stimulation their toddler gets at home renders outside intellectual stimulation superfluous and are concerned only about a pleasant play environment.

To assess such advice, we need to ask how the evidence was collected and what it can establish. The same questions matter for public programs: a plausible account of why an intervention should work is a reason to test it.

A Continuum of Convincingness

The CNN and CTV reports appear to conflict, but preventing infection and studying microbial exposure in immune development are different questions. Consider what kinds of evidence could help distinguish them. The comparisons below are useful for examining study design; they do not establish a safe exposure regimen for an infant.

In a natural experiment, an event, rule, or other process outside the researcher’s control creates exposure differences that can support a causal comparison. The chapter uses the term more broadly for several observational comparisons. Those comparisons can be informative, but similar-looking groups may differ in other ways that explain the outcome.

Suppose you knew that East Germans were less likely to have allergies than West Germans. Suppose you knew that Russians were less likely to have allergies than Finns. Suppose you knew that farmers were less likely to have allergies than city dwellers. Suppose you knew that children who had attended day care were less likely to have allergies than children who didn’t. Suppose you knew that children who had pets when they were infants were less likely to have allergies than children who didn’t. Suppose you knew that children who had lots of diarrhea as infants were less likely to have allergies than children who had less diarrhea. Suppose you knew that infants who were born vaginally were less likely to have allergies than infants who were born by Caesarian section.

These are the associations reported in the studies discussed here. 1 The groups differ in exposures that might help explain allergy outcomes, but they can also differ in diet, housing, access to care, and other relevant factors. Taken together, the comparisons can motivate tests of hypotheses about microbial exposure and immune development. Allergy and asthma involve immune responses, but describing them generally as autoimmune diseases is inaccurate. Autoimmunity involves an immune response against the body’s own tissues; an allergy is a mistakenly “protective” response to an otherwise harmless substance.

Allergies range from annoying to debilitating in their effects, and asthma can be lots worse than that. Every day in the United States tens of thousands of kids miss school because of asthma, hundreds of people enter the hospital because of it, and some people die of it.

One hypothesis about the historical East–West comparisons was that differences in microbial exposure contributed to the reported differences in allergy rates. Country labels alone, however, cannot measure hygiene or separate it from other differences. (Interestingly, many years ago a Polish immigrant to the United States told me half-jokingly that he thought allergies were an American invention. He may have been onto something.)

We can also assume that children raised on farms are more likely to be exposed to a wide variety of bacteria than children who grow up in cities. We know that children with pets are exposed to a wider range of bacteria—including fecal bacteria—than those who don’t have pets. We know that toddlers are walking petri dishes who expose one another to a wider range of bacteria in pre-K programs than they would encounter if they just stayed at home. Lots of diarrhea can be the result of exposure to lots of bacteria. Vaginal birth exposes the infant to the full panoply of bacteria in the mother’s vagina.

These comparisons support further investigation of microbial exposure, rather than a general conclusion that bacteria are beneficial or that deliberate exposure to pathogens is safe. A study discussed here also used infant rectal swabs to examine whether bacterial diversity predicted later immune outcomes at the age of six. 2 That is correlational, or observational, evidence. A predictive relationship does not by itself identify the cause or an appropriate intervention.

A hypothesis called the “germ exposure theory” was proposed to account for these observational patterns. Early exposure to bacteria could be expected to stimulate the immune system and such stimulation might have beneficial effects down the road. The young immune system might be strengthened in ways that allow it to adapt and regulate itself, with possible effects on later inflammatory and allergic responses. A plausible mechanism strengthens the case for research, but does not establish the safety or benefit of a particular exposure.

Natural experiments, correlational evidence, and plausible theories are all well and good. But I would want to see a true experiment of the double-blind, randomized control sort, with babies assigned by the proverbial flip of a coin to an experimental high-bacteria exposure condition versus a control , low-bacteria condition. Both the experimenter and the participants (the mothers in this case) should be ignorant of (blind to) the condition the babies were assigned to. Blinding reduces the risk that expectations, differences in care, or outcome assessment will be influenced by knowledge of the assigned condition. An ethically acceptable experiment would need a defined intervention, safety safeguards, and meaningful outcomes. Animal models can help investigate biological mechanisms before human research, but an effect in another species cannot simply be presumed to hold in humans.

Researchers have studied the effects of bacteria exposure in young mice. 3 Instead of exposing some of the mice to extra-high levels of bacteria, the scientists went the other direction and created a germ-free environment for some of the mice and left control mice in normal lab-mouse conditions, which, trust me, are decidedly not germ-free. The germ-free mice developed abnormal levels of a type of killer T cell in parts of the colon and in the lungs. These T cells were subsequently recruited to attack even nonthreatening substances, with resulting inflammation, allergy, and asthma.

The mouse experiment adds evidence about a possible mechanism. It does not establish that deliberately exposing an infant to dirt, feces, or pathogens is beneficial, nor does it resolve the apparent conflict between the news reports.

The week I finished writing these paragraphs, an article in JAMA Pediatrics reported a study using five drops of a solution containing the bacterium Lactobacillus reuteri. 4 It reported a nearly 50 percent reduction in colic-related crying. That is evidence about a specific studied intervention and outcome, rather than a general endorsement of bacterial exposure.

A different research question concerns associations between antibiotic use and later illness. What if you knew that the richer the country, the higher the rate of inflammatory bowel disease (IBD), including Crohn’s disease and ulcerative colitis? 5 These diseases can be very serious, even fatal. They can cause abdominal pain, vomiting, diarrhea, rectal bleeding, severe internal cramps, anemia, and weight loss. IBD involves chronic inflammation and immune dysfunction. The association with national wealth is observational: healthcare access, diagnosis, environment, and other factors may contribute to it.

People of a certain age will remember that their childhood was plagued with middle-ear infections whereas their own children, courtesy of amoxicillin, got rid of them almost as soon as they got them. The richer the country, of course, the more likely you are to visit a doctor, to have antibiotics prescribed, and to be able to fill the prescription with insurance money or your own money. But if you’re like me, you may have wondered whether it was a great idea to get all those antibiotics. The associations discussed here give researchers reasons to investigate that concern.

Kids who have lots of ear infections and lots of antibiotics are more likely to develop IBD later in life. 6 Antibiotics are overeager. They kill the good, the bad, and the ugly among the microorganisms in the gut. Antibiotic use even in adulthood appears to be associated with subsequent bowel disease. Researchers have found that adults who have IBD are twice as likely to have had multiple antibiotic prescriptions in the previous two years. 7

The association between antibiotics and IBD is still observational. A separate experimental question is whether changing the gut microbial community can improve symptoms. If altered gut microbial communities contribute to some forms of IBD, transferring material from a carefully screened donor offers a hypothesis to test. The proposed mechanism alone cannot establish that such treatment is safe or effective.

Brave scientists, and even braver patients, tried the experiment. (“So, Ms. Jones, in this experiment we’re going to pump the contents of a stranger’s intestines into yours. And not just because it’s never been done, but because there’s a possibility it may be good for you.”)

In the experiment described here, treated patients were more likely to improve than control subjects who received saline solution. Interpreting that result requires the particular diagnosis, treatment protocol, outcomes, and safety evidence. Neither this result nor an association between antibiotics and illness determines whether an individual infection should be treated.

From Natural to Proper Experiments

Natural experiments can have tremendously important implications that beg to be studied by proper experiments. Children of parents with little education, and who are therefore at risk for low academic achievement themselves, are likely to have a poor elementary school outcome if their first-grade teacher, judged by observers, is in the bottom third of teaching effectiveness. If they’re lucky enough to get a teacher in the top third of effectiveness, their performance is likely to nearly equal the performance of middle-class children. 8 This finding constitutes a natural experiment. If children were to be randomly assigned to classrooms with teachers of different judged competence, we would have a true experiment. Meanwhile, what parent would be indifferent to teacher effectiveness after hearing about the result of the natural experiment?

Greenery in a city is nice. Nicer than you might assume, actually. A study of identical public housing apartment dwellings in Chicago found about half as many reported crimes in apartment houses surrounded by greenery as in apartment houses surrounded by barren land or concrete. 9 In light of the sort of subtle situational cues that can profoundly affect behavior, discussed in Chapter 1, this is not such a surprising finding.

The study is probably a real experiment because housing officials in Chicago believe that assignments to a particular project are made randomly—and there’s no reason to assume that’s not correct. On the other hand, laypeople don’t necessarily mean the same thing by the term “random” as scientists do, so complete confidence in the greenery/low crime hypothesis awaits a study with verifiably random assignment to dwelling to test whether the relationship between greenery and crime is causal rather than explained by other differences between the dwellings or residents. Obviously, such an experiment is sorely needed. If the true experiment findings duplicate the natural experiment findings, a cost-benefit analysis of the kind discussed in Chapter 4 would be badly needed. Such a study would examine the effects of ripping up concrete and putting in trees and weigh them against the cost in dollars. The analysis might show that the landscape change is a bargain for a city.

Scientists often get their ideas when they realize that some observation they’ve made constitutes a natural experiment. The eighteenth-century physician Edward Jenner noticed that milkmaids rarely got smallpox, a disease related to the cowpox to which the milkmaids would have been exposed. Maybe milkmaids were less likely to get smallpox than butter churners because cowpox somehow protected against smallpox. Jenner found a young milkmaid with cowpox on her hand and inoculated an eight-year-old boy with some material from it. The boy developed a fever and discomfort in his armpits. Later, Jenner inoculated the boy with smallpox from a lesion from a smallpox sufferer. The boy did not develop the disease, and Jenner correctly announced that he had discovered a treatment that could prevent smallpox. The Latin word for cow is vacca, and Latin for cowpox is vaccinia, so Jenner called his treatment vaccination. A natural experiment led to a proper experiment, and the results changed the world for the better. Smallpox was eventually eradicated through vaccination. Jenner’s historical experiment illustrates a move from an observation to an intervention, though its procedures would require a very different ethical assessment today.

The High Cost of Not Doing Experiments

We can pay dearly, in blood, treasure, and well-being, for experiments that aren’t done.

At the time of writing, Head Start had been in existence for nearly fifty years, with reported spending of $200 billion. Head Start is a preschool program for poor, primarily minority children intended to improve their health, academic achievement, and, it was hoped, their IQs. What have we gotten for our investment? The program did improve the children’s health, and initially improved IQ and academic success. But the cognitive gains lasted only a few years; by mid–elementary school the children were doing no better than children who hadn’t been in the program.

We don’t know for sure whether the Head Start children fared any better as adults than did children who weren’t in the program. 10 That’s because assignment to the program was not random. Kids who ended up in Head Start could have differed in any number of unknown ways from those who didn’t attend the program. All the adult outcome data, of which there is shockingly little, rely on purely retrospective information about assignment. People had to remember whether they had been in a preschool program and if so, which one. Retrospective studies are subject to a great deal of potential error, especially when the memories in question go back to events decades in the past.

The retrospective studies do show apparent gains in life outcomes in adulthood for children who were in Head Start. 11 But this result doesn’t even come up to the level of a natural experiment because it would be surprising if there weren’t preexisting differences between children who were in Head Start and those who were not. A lot of money continues to be spent on something that may or may not be effective.

Fortunately, as you’ll recall from Chapter 4, we know some preschool programs have a huge effect on adult outcomes. Randomized-assignment experiments with programs more intensive than Head Start produced modest IQ gains that were long-lasting, but, much more important, academic improvements and economic gains for adults who had been in the treatment groups were huge. The costs of not knowing what works and what doesn’t in the way of preschool programs have been very great indeed. The $200 billion for Head Start might have been better spent on a smaller number of particularly vulnerable children, providing them with more intensive experiences. That might have produced far greater societal benefits. (And we do in fact know that the poorer the child, the greater the impact of high-quality early childhood education. It doesn’t seem to much affect outcomes for middle-class children.) 12

Moreover, no experiments were conducted to find out what aspects of Head Start (if any) were the most effective. Is it better to focus on academics or on social factors? Half days or full days? Are two years needed or would one year make almost as much difference? The social and economic consequences of knowing the answers to such questions would be enormous. And getting the answers would have been easy and dirt cheap in comparison to what has been spent.

At least it’s unlikely that Head Start does any harm to children who participate in it. But many interventions dreamed up by nonscientists actually do harm. Well-intentioned people invented a program to help possible trauma victims soon after a tragedy has occurred. So-called grief counselors encourage participants in a treatment group to recount the incident from their own perspective, describe their emotional responses, offer their comments on others’ reactions, and discuss their stress symptoms. The counselor assures participants that their reactions are normal and that such symptoms generally diminish with time.

Some nine thousand grief counselors descended on New York City in the wake of 9/11. Grief counseling of this sort seems like an excellent idea to me. However, behavioral scientists have conducted more than a dozen randomized experiments examining critical incident stress debriefing (CISD). They have found no evidence that the activity has a positive effect on depression, anxiety, sleep disturbance, or any other stress symptoms. 13 There is some evidence that people who undergo CISD are more likely to experience full-blown traumatic stress disorder. 14

As it happens, behavioral scientists have found some interventions that actually are effective for trauma victims. A few weeks after a critical incident, the social psychologist James Pennebaker has trauma victims write down, in private, and for four nights in a row, their innermost thoughts and feelings about the experience and how it affects their lives. 15 And that’s all. No meetings with a counselor, no group-therapy encounters, no advice about how to handle the trauma. Just a writing exercise. The studies discussed here reported benefits from this writing exercise.

It’s not at all plausible to me that this exercise would be very effective. Certainly not as plausible as the idea that immediate intervention, grief sharing, and advice would be effective. But there it is. Assumptions need testing.

Pennebaker thinks his writing exercise works because it helps people, after a period of suffering and incubation, to develop a narrative to understand the event and their reactions to it. And it seems to be the case that the people who improve most are those who began the exercise with inchoate and disorganized descriptions and ended with coherent, organized narratives that gave meaning to the event.

Other well-meaning people have tried to inoculate teenagers against peer pressure to commit crimes and engage in self-destructive behavior, with results that are sometimes even more disappointing than CISD for trauma victims. Decades ago inmates in Rahway State Prison in New Jersey decided to do something to warn at-risk adolescents of the dire consequences of criminal behavior. The inmates showed the kids what prison was like, including graphic accounts of rape and murder within its walls. An award-winning documentary on the Arts and Entertainment (A&E) channel christened the program Scared Straight. The name and the practice spread widely throughout the United States.

Do Scared Straight programs work? Seven experimental tests of the programs have been carried out. 16 Every single study found the Scared Straight kids to be more likely to commit crimes than kids in the control group who were exposed to no intervention at all. On average, the increase in criminal activity was about 13 percent.

At the time of writing, the Rahway program still existed, and more than fifty thousand East New Jersey kids had reportedly passed through it. Let’s multiply fifty thousand by 13 percent. The figure we get is sixty-five hundred. That arithmetic does not establish how many additional crimes occurred. A relative increase in offending needs a baseline rate, and an average effect from several studies cannot automatically be applied to every participant in this particular program. And that’s just one area of New Jersey. The program has been duplicated in many other communities. A study commissioned by the Washington State Institute for Public Policy estimated that every dollar spent on Scared Straight incurs crime and incarceration costs of more than two hundred dollars.

Why doesn’t Scared Straight work? It certainly seems to me that it should. We don’t know why it doesn’t work, and we certainly don’t know why it should be counterproductive, but that doesn’t matter. It’s a tragedy that it was invented and a crime that it hasn’t been stopped.

Why hasn’t it been stopped? I’ll venture the guess that it just seems so obvious that it should work. Many people, including many politicians, prefer to trust their intuitively compelling causal hypotheses over scientific data. It doesn’t help that scientists can’t offer any convincing explanations for why Scared Straight doesn’t work. Scientists can also cling to intuitive causal theories despite conflicting data. The safeguard is a willingness to revise those theories when tests contradict them: ATTBW, Assumptions Tend to Be Wrong. (As of this writing, the A&E channel is still airing a program singing the praises of Scared Straight.)

D.A.R.E. is another elaborate attempt to keep kids out of trouble. As part of the Drug Abuse Resistance Education program, local police officers undergo eighty hours of training in teaching techniques and then visit classrooms to present information intended to reduce drug, alcohol, and tobacco use. The funding figure cited at the time of writing was $1 billion per year from state, local, and federal sources. The website then reported participation by 75 percent of American school districts and forty-three countries. The evaluations discussed here concerned versions used over the preceding thirty years and did not find decreased drug use. 17 My criticism concerns those versions and the evidence available in that period; replacement programs require their own evaluations.

Why doesn’t D.A.R.E. work? We don’t know. It would be nice if we did, but causal explanations are unnecessary. As it happens, some programs intended to lower the likelihood of drug, alcohol, and tobacco use do work. These include LifeSkills Training and the Midwestern Prevention Project. 18 These programs have elements missing from the original D.A.R.E. program, notably teaching preadolescents skills in resisting peer pressure. The inventors of D.A.R.E. made the assumption that police are important social influence agents for teenagers. A social psychologist could have told them that peers are a much more effective source of influence.

The more successful programs also provide information about drug and alcohol use among teenagers and adults. Recall that such information often surprises because these rates are lower than most youngsters believe, and accurate knowledge about others’ behavior can lower rates of abuse. 19

Meanwhile, programs that damage young people are still being conducted, and programs that help are underused or used not at all. These historical cases show why programs need evaluation even when their rationale seems persuasive.

Summing Up

Observational comparisons can suggest useful hypotheses. A natural experiment offers stronger causal evidence when the process that creates the comparison groups credibly separates the exposure from other influences. Ordinary differences between countries or families need more caution.

Random assignment makes treatment independent of preexisting characteristics in expectation; it does not guarantee identical groups in a particular sample. A well-designed trial can support a causal estimate, but chance imbalance, attrition, noncompliance, measurement error, and other problems still matter. Blinding helps reduce biases arising from knowledge of treatment assignment.

The chapter’s historical account of Head Start asks whether the reported $200 billion investment produced lasting benefits and whether different program designs would have worked better. Its examples of preschool education, Scared Straight, D.A.R.E., and critical incident stress debriefing illustrate the need to evaluate specific interventions. Findings about one version of a program, or one kind of trauma intervention, do not settle every related approach.

Testing can reveal benefits or harms that a compelling explanation alone would miss. Public decisions improve when evaluations are designed well, results are examined critically, and programs are revised in response.