Rejecting the Alternative

Elementary statistics makes the front page of the New York Times
Michael Kinsley take notice.


"Witth his condition deteriorating from Parkinson's disease last year, Steve Kaufman gave up making improvements to his home in Algonquin, Ill. "I couldn't even hold a nail stable," he recalled.

Earlier this year, after taking an experimental drug in a clinical trial, Mr. Kaufman built new kitchen cabinets and an outdoor deck. He was so steady he could walk across a narrow piece of lumber like an Olympic gymnast on the balance beam."

But

The drug, however, is no longer available to Mr. Kaufman or other Parkinson's patients in clinical trials. In June, its developer, Amgen, announced that the drug, which is called glial cell line-derived neurotrophic factor, or GDNF, had not proved better than a placebo.


"To control for the placebo effect, Amgen began a 34-patient trial last year in which all the patients had the pumps and catheters implanted but for the first six months half got GDNF and the other half saline solution. The results, announced in June, showed that while the group getting GDNF did somewhat better than the control group on a scale measuring the severity of symptoms, the difference in the results was not statistically significant."

Let us review the two claims in bold, both due to AMGEN the manufacturer of GDNF. In the first it is asserted that the treated group did not do better than the group given the placebo. In the second it is asserted that they did do better but that the difference was not statistically significant.

The statistically insignificant positive effect is considered proof that the true effect is zero. This is an exceedingly common but blatent error due to incomprehension of the meaning of the phrase "statisticially insignificant". Another approach would be to perform a larger trial in order to get more precise estimates and find out if the effect of the treatment is positive as weakly suggested by the initial trial.

Note the number in bold -- 34. Amgen appears to have decided that if a trial with a sample size of 34 doesn't show a significant effect, there is no reason to look further.

Now let's consider the cost of type I and type II error in this case. Type II error (failure to reject the nul that GDNF does not work even though it does) means people with Parkinson's suffer horrible disability and early death. Type one error means that a potentially large number of un-necessary surgery involving drilling holes in peoples skull will be performed until it becomes clear that the treatment doesn't work. Both are costly, but in this case, type II error is even worse, so rejecting the alternative is worse than a mistake, it is a crime.

Of course a larger study would be expensive "Amgen had spent hundreds of millions of dollars over 10 years on the drug's development." This is an absurdly tiny sum compared to the reasonable chance of a useful treatment for Parkinson's. Recall the voters of California just voted for 3 billion on a long shot.

And the point estimate was positive! It boggles the mind.