(August 29th, 2014) It took one year and thousands of statistical re-analyses but now a headline-grabbing paper on functional genomics of human well-being has been debunked as nothing but “artifacts of dubious analyses and erroneous methodology”.
There are many roads to happiness. Opening a one-litre box of chocolate ice cream and feasting, striving toward meaning or publishing a paper in PNAS and attracting huge media attention. Last year, US scientists including Barbara Frederickson (University of North Carolina) and Steve Cole (University of California, Los Angeles) increased their psychological well-being by succeeding in the latter option. Recently, though, their joy was tempered when an international team of researchers found that all their results are devoid of any meaning.
Coincidentally, the PNAS authors’ paper was about happiness. They wanted to explore the “biological implications of hedonic [seeking pleasure] and eudaimonic [striving for meaning] well-being through the lens of the human genome - a system of ~ 21,000 genes that has evolved fundamentally to help humans survive and thrive (i.e., be well)”. Indeed, their analysis led them to believe there was a difference. Peripheral blood mononuclear cells from pleasure seekers showed “increased expression of proinflammatory genes and decreased expression of genes involved in antibody synthesis and type I IFN response”. Intriguingly, these same gene expression profiles can also be observed in people suffering from chronic stress. So, for a happy, long life: no more ice cream and cigarettes?
Nah, don’t worry. The results are probably wrong. Immediately after the study’s publication, James Coyne, University of Groningen, noticed there was something fishy and blogged about it. More and more colleagues aligned with him and started to anatomise the article. Coyne writes in a PLoS blog entry, “The team gave an extraordinarily careful look at the article, noting its fuzzy theorizing and conceptual deficiencies, but we did much more than that. We obtained the original data and asked the authors of the original paper about their complex analytic methods. We then reanalyzed the data, following their specific advice. We tried alternative analyses and even re-did the same analyses with randomly generated data. Overall, our hastily assembled group performed and interpreted 1000s of analyses, more than many productive labs do in a year.” All their re-analysing efforts were not in vain. Recently, PNAS published their “Critical reanalysis of the relationship between genomics and well-being”.
Besides statistical flaws, their critic also points to methodological weaknesses. One of them is the small sample size, which consisted of only 80, mostly white adults. The actual sample size should have been even smaller, as vital information like control gene data was missing for four participants. What’s more, the US scientists only obtained one blood sample from each participant. Coyne also criticised the choice of genes to be analysed. “The choice of 53 human genes out of some 20,000 potential candidates appears to be based almost exclusively on the prior opinion of one of the authors (…) The decision to apply a weighting of exactly 1.00 or -1.00 to the regression coefficients associated with each of these genes, thus assigning an identical magnitude of effect to every gene, seems to be completely arbitrary”.
“We believe”, Coyne and colleagues write in conclusion, “it is vitally important that researchers, practitioners and laypersons alike should be made aware of the deficiencies of their study and, in particular, of their statistical analyses, which appear to have conjured non-existent effects out of thin air. For positive psychology to move into the health arena in a responsible way, the research, on which advice is based, must be rigorous and must avoid giving false hope or even causing iatrogenic harm. It seems to us that Fredrickson et al.’s study falls far short of the level of responsible scholarship required in promoting human health and well-being.”
According to Coyne, Frederickson and Cole already replied, attacking his methodology and calling it inappropriate. It might well be a long time before happiness returns to both parties.