KSJ Tracker August 14, 2008

Wall Street Journal: Reporter reviews data, says company published flawed research analysis

Reporters usually aren't technical experts. They talk to experts and cite their expertise, usually by name. But sometimes one will stand on his or her own two feet and write important declarations in a strong voice attributed to nobody else. Everybody in The Tracker's normal readership who knows what a Wald interval is, raise your hands. If you did, you recognize the notation to the upper right. Wall St. Journal Reporter Keith J. Winstein - an MIT E.E. grad - apparently knows enough about it to say it's the basis for a flawed statistical equation. It sees significance where there may be none, he writes today.  He baldly calls Boston Scientific Corp. out for, he asserts, improperly using it in a tussle with the FDA. The company sells cardiac stents. Approval of one big money maker is justified primarily on the company's analysis of clinical trial data. The medical device company's published study, Winstein reports, has a p-value that translates to a 4.85 percent chance that the data's suggested correlations are just flukes. That's just short of 5 percent, or flunking.  "The Journal's calculations" (using better tools than a Wald interval) indicate a more robust p-value is 5.1 percent, he writes. That could, presumably, lead the FDA to regard the stents as no better than those already on the market, hence no permit. The company says it followed s.o.p. for such studies and did nothing amiss. This is remarkable. Many reporters, medical reporters in particular, try to master the rudiments of statistics, at least enough to recognize a p-value that is persuasive, and to understand the perils of a small n in the denominator. But that's typically about it (is for me anyway). This will be fascinating to follow. Eventually, one would like to know if the journal (presumably meaning Winstein and a few consultants)  really did the math itself or if the essence of the story came in from an outside whistle blower. If he dove into this statistical labyrinth on his own and emerged crying foul ... well, that is not the sort of thing one sees in the news room every day. At a site affiliated with WebMD and called TheHeart, an interesting discussion of the Journal article is already up, in a piece by Shelley Wood.  The comments, largely from physicians, suggest a review and perhaps overhaul of medical statistical standards is due.



comments powered by Disqus