Donna Gordon Blankinship of the AP wrote a story yesterday in which she did a wonderful job of explaining why she should not have written the story.
The study was about children's TV watching, and her lede was, "Teaching parents to switch channels from violent shows to educational TV can improve preschoolers' behavior, even without getting them to watch less, a study found." She goes on to write that "the results were modest and faded over time, but may hold promise for finding ways to help young children avoid aggressive, violent behavior, the study authors and other doctors said." Or not–a study with modest results that quickly faded probably doesn't offer promise of anything.
Then she explains how the study was done. Parents filled out diaries recording what their children watched and how their children behaved. Some were told to encourage the children to watch "Sesame Street" and similar shows, and to avoid such shows as "Power Rangers." At six months, both groups showed improved behavior, but there was "a little bit more improvement" in the group coached to watch less violent shows. At a year, there was no difference.
"The study has some flaws," Blankinship writes. "The parents weren't told the purpose of the study, but the authors concede they probably figured it out and that might have affected the results."
The improvement in behavior was only "a little bit" even when parents had figured out that's what they were supposed to look for.
The evidence to support the lede was awful. The lede should have been something like, "A study designed to see whether watching less violent TV shows would affect children's behavior didn't find any important difference."
The problem with this study is that most of us want it to be right. The New York Times fell into the same trap. Researchers "found that the experiment reduced the children’s aggression toward others, compared with a group of children who were allowed to watch whatever they wanted," Catherine Saint Louis wrote. She quotes a researcher who says the public health impact "could be very meaningful." Based on what? She doesn't ask him why he concludes that from a study with such weak evidence.
Let's try a thought experiment. Suppose the study had found that watching "Power Rangers" had much better effects on children's behavior than "Sesame Street." Reporters would have been all over the study, highlighting its weaknesses. The producers of Sesame Street would have put out a statement denouncing the research. A Congressional panel would have investigated. Pitchfork-carrying mothers would have descended on the homes of the researchers…
OK, maybe not all of that. But the reporters who covered the story would clearly, in my guess, have been more critical if the study hadn't found the (poorly supported) result we wanted it to find.
The lesson is we always need to be wary of being too accepting of stories that accord with our preconceived ideas.
Who do you want your kids to emulate? Elmo, or the pink Power Ranger? I'd go for Super Grover. He has a big heart. And I have no idea how he affects children's behavior.
-Paul Raeburn
Leave a Reply