False positives: fraud and misconduct are threatening scientific research

irk Smeesters had spent several years of his career as a social psychologist at Erasmus University in Rotterdam studying how consumers behaved in different situations. Did colour have an effect on what they bought? How did death-related stories in the media affect how people picked products? And was it better to use supermodels in cosmetics adverts than average-looking women?

The questions are certainly intriguing, but unfortunately for anyone wanting truthful answers, some of Smeesters’ work turned out to be fraudulent. The psychologist, who admitted “massaging” the data in some of his papers, resigned from his position in June after being investigated by his university, which had been tipped off by Uri Simonsohn from the University of Pennsylvania in Philadelphia. Simonsohn carried out an independent analysis of the data and was suspicious of how perfect many of Smeesters’ results seemed when, statistically speaking, there should have been more variation in his measurements.

The case, which led to two scientific papers being retracted, came on the heels of an even bigger fraud, uncovered last year, perpetrated by the Dutch psychologist Diederik Stapel. He was found to have fabricated data for years and published it in at least 30 peer-reviewed papers, including a report in the journal Science about how untidy environments may encourage discrimination.

The cases have sent shockwaves through a discipline that was already facing serious questions about plagiarism.

“In many respects, psychology is at a crossroads – the decisions we take now will determine whether or not it remains a serious, credible, scientific discipline along with the harder sciences,” says Chris Chambers, a psychologist at Cardiff University.

“We have to be open about the problems that exist in psychology and understand that, though they’re not unique to psychology, that doesn’t mean we shouldn’t be addressing them. If we do that, we can end up leading the other sciences rather than following them.”

Cases of scientific misconduct tend to hit the headlines precisely because scientists are supposed to occupy a moral high ground when it comes to the search for truth about nature. The scientific method developed as a way to weed out human bias. But scientists, like anyone else, can be prone to bias in their bid for a place in the history books.

Increasing competition for shrinking government budgets for research and the disproportionately large rewards for publishing in the best journals have exacerbated the temptation to fudge results or ignore inconvenient data.

Massaged results can send other researchers down the wrong track, wasting time and money trying to replicate them. Worse, in medicine, it can delay the development of life-saving treatments or prolong the use of therapies that are ineffective or dangerous. Malpractice comes to light rarely, perhaps because scientific fraud is often easy to perpetrate but hard to uncover.

The field of psychology has come under particular scrutiny because many results in the scientific literature defy replication by other researchers. Critics say it is too easy to publish psychology papers which rely on sample sizes that are too small, for example, or to publish only those results that support a favoured hypothesis. Outright fraud is almost certainly just a small part of that problem, but high-profile examples have exposed a greyer area of bad or lazy scientific practice that many had preferred to brush under the carpet.

Many scientists, aided by software and statistical techniques to catch cheats, are now speaking up, calling on colleagues to put their houses in order.

Those who document misconduct in scientific research talk of a spectrum of bad practices. At the sharp end are plagiarism, fabrication and falsification of research. At the other end are questionable practices such as adding an author’s name to a paper when they have not contributed to the work, sloppiness in methods or not disclosing conflicts of interest.

“Outright fraud is somewhat impossible to estimate, because if you’re really good at it you wouldn’t be detectable,” said Simonsohn, a social psychologist. “It’s like asking how much of our money is fake money – we only catch the really bad fakers, the good fakers we never catch.”

If things go wrong, the responsibility to investigate and punish misconduct rests with the scientists’ employers, the academic institution. But these organisations face something of a conflict of interest. “Some of the big institutions … were really in denial and wanted to say that it didn’t happen under their roof,” says Liz Wager of the Committee on Publication Ethics (Cope). “They’re gradually realising that it’s better to admit that it could happen and tell us what you’re doing about it, rather than to say, ‘It could never happen.'”

There are indications that bad practice – particularly at the less serious end of the scale – is rife. In 2009, Daniele Fanelli of the University of Edinburgh carried out a meta-analysis that pooled the results of 21 surveys of researchers who were asked whether they or their colleagues had fabricated or falsified research.

Publishing his results in the journal PLoS One, he found that an average of 1.97% of scientists admitted to having “fabricated, falsified or modified data or results at least once – a serious form of misconduct by any standard – and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% for falsification, and up to 72% for other questionable research practices.”

A 2006 analysis of the images published in the Journal of Cell Biology found that 1% of accepted papers have at least one image that has been manipulated in a way that affects the interpretation of the data – though the authors made no conclusions about intent.

Rise in retractions

According to a report in the journal Nature, published retractions in scientific journals have increased around 1,200% over the past decade, even though the number of published papers had gone up by only 44%. Around half of these retractions are suspected cases of misconduct.

Wager says these numbers make it difficult for a large research-intensive university, which might employ thousands of researchers, to maintain the line that misconduct is vanishingly rare.

New tools, such as text-matching software, have also increased the detection rates of fraud and plagiarism. Journals routinely use these to check papers as they are submitted or undergoing peer review. “Just the fact that the software is out there and there are people who can look at stuff, that has really alerted the world to the fact that plagiarism and redundant publication are probably way more common than we realised,” says Wager. “That probably explains, to a big extent, this increase we’ve seen in retractions.”

Ferric Fang, a professor at the University of Washington School of Medicine and editor in chief of the journal Infection and Immunity, thinks increased scrutiny is not the only factor and that the rate of retractions is indicative of some deeper problem.

He was alerted to concerns about the work of a Japanese scientist who had published in his journal. A reviewer for another journal noticed that Naoki Mori of the University of the Ryukyus in Japan had duplicated images in some of his papers and had given them different labels, as if they represented different measurements. An investigation revealed evidence of widespread data manipulation and this led Fang to retract six of Mori’s papers from his journal. Other journals followed suit.

Self-correction

The refrain from many scientists is that the scientific method is meant to be self-correcting. Bad results, corrupt data or fraud will get found out – either when they cannot be replicated or when they are proved incorrect in subsequent studies – and public retractions are a sign of strength.

That works up to a point, says Fang. “It ended up that there were 31 papers from the [Mori] laboratory that were retracted, many of those papers had been in the literature for five-10 years,” he says. “I realised that ‘scientific literature is self-correcting’ is a little bit simplistic. These papers had been read many times, downloaded, cited and reviewed by peers and it was just by the chance observation by a very attentive reviewer that opened this whole case of serious misconduct.”

Extraordinary claims that change the paradigm for a field will elicit lots of attention and people will look at the results very carefully. But cases such as Dr Mori’s – where work is flawed and falsified but the results themselves are not particularly surprising or sensational and may even corroborated by others who perform their experiments legitimately – the misconduct is difficult to detect. “It’s not that the results are wrong, it’s that the data are false,” says Fang.

And, often, research studies are very difficult to replicate. “If someone says they did a 15-year clinical study with 9,000 subjects and they publish their results, you may have to take their word for it because you’re not going to be able to run out and recruit 9,000 patients of your own and do a 15-year study just to try to corroborate something that somebody else has done,” says Fang. “A number of cases recently have come to light only because the investigators didn’t have institutional review board approval for their studies. Upon digging deeper, the institutions questioned whether any of the studies were done at all. This kind of misconduct is very difficult to detect otherwise.”

Selective publishing

In psychology research, there is a particular problem with researchers who selectively publish some of their experiments to guarantee a positive result. “Let’s say you have this theory that, when you play Mozart, people want to pay more for musical instruments,” says Simonsohn. “So you do a study and you play Mozart (or not) and you ask people, ‘How much would you pay for a piano or flute and five instruments?'”

If it turned out that only the price of a single type of instrument, violins, say, went up after people had listened to Mozart, it would be possible to publish a research paper that omitted the fact that the researchers had ever asked about any other instruments. This would not allow the reader to make a proper assessment of the strength of the effect that Mozart may (or may not) have on how much a person would pay for musical instruments.

Fanelli has examined this positive result bias. He looked at 4,600 studies across all disciplines between 1990 and 2007, and counted the number of papers that, after declaring an intent to test a particular hypothesis, reported a positive support for it. The overall frequency of positive supports had grown by more than 22% over this time period. In a separate study, Fanelli found that “the odds of reporting a positive result were around five times higher among papers in the disciplines of psychology and psychiatry and economics and business compared with space science”.

Culture of neophilia

This issue is exacerbated in psychological research by the “file-drawer” problem, a situation when scientists who try to replicate and confirm previous studies find it difficult to get their research published. Scientific journals want to highlight novel, often surprising, findings. Negative results are unattractive to journal editors and lie in the bottom of researchers’ filing cabinets, destined never to see the light of day.

“We have a culture which values novelty above all else, neophilia really, and that creates a strong publication bias,” says Chambers. “To get into a good journal, you have to be publishing something novel, it helps if it’s counter-intuitive and it also has to be a positive finding. You put those things together and you create a dangerous problem for the field.”

When Daryl Bem, a psychologist at Cornell University in New York, published sensational findings in 2011 that seemed to show evidence for psychic effects in people, many scientists were unsurprisingly sceptical. But when psychologists later tried to publish their (failed) attempts to replicate Bem’s work, they found journals refused to give them space. After repeated attempts elsewhere, a team of psychologists led by Chris French at Goldsmith’s, University of London, eventually placed their negative results in the journal PLoS One this year.

There is no suggestion of misconduct in Bem’s research but the lack of an avenue in which to publish failed attempts at replication suggests self-correction can be compromised and people such as Smeesters and Stapel can remain undetected for a long time.

In some cases, misconduct (or fraud) has grave implications. In 2006, Anil Potti and colleagues at Duke University reported in the New England Journal of Medicine that they had developed a way to track the progression of a patient’s lung cancer with a device, called an expression array, that could monitor the activity of thousands of different genes. In a subsequent report in Nature Medicine, the same scientists wrote about a way to use their expression array to work out which drugs would work best for individual patients with lung, breast or ovarian cancer, depending on their patterns of gene activity. Within months of that publication, the biostatisticians Keith Baggerly and Kevin Coombes of the MD Anderson Cancer Centre in Houston had their doubts, and began uncovering major flaws in the work.

“It looked so promising that they actually started to do trials of cancer patients, they chose the chemotherapy depending on this test,” says Wager. “The test has turned out to be completely invalid, so people were getting the wrong therapy, because the paper was not retracted quickly enough.”

Blowing the whistle

Despite Baggerly and Coombes raising the alarm several times with the institutions involved, it was not until 2010 that Potti resigned from Duke University and several of the papers referring to his work on the expression array were retracted.”Usually there is no official mechanism for a whistleblower to take if they suspect fraud,” says Chambers. “You often hear of cases where junior members of a department, such as PhD students, will be the ones that are closest to the coalface and will be the ones to identify suspicious cases. But what kind of support do they have? … That’s a big issue that needs to be addressed.”

In July this year, a group of the UK’s main research funders and university groups published a Concordat to Support Research Integrity. “I don’t think anyone would want to see a command-control direct regulation approach here,” says Christopher Hale, deputy director of policy at Universities UK. “The concordat … outlines a framework and then identifies how people fit within that and what actions they will take forward to strengthen it.” The concordat requires institutions to have a process in place for dealing with misconduct, which includes appointing a senior person at the institution who can provide the necessary leadership and oversight during investigations.

Michael Farthing, vice-chair of the UK Research Integrity Office and vice-chancellor of the University of Sussex, has been a long-time campaigner on getting institutions and funders to take research misconduct seriously. In a recent article for Times Higher Education, Farthing said he supported the concordat but that it would not be enough. He stopped short of suggesting a statutory regulator for research but wrote: “Government and research leaders should take action to support and encourage excellence in research integrity, not sit on their hands until – as has happened in other countries – a scandal drives them towards legislation.”

Statements of principle are one thing – every university and research council probably already has one applauding honourable research and deploring fraud – the key is the steps institutions take in understanding and de-incentivising misconduct.

The economics of science

The pressure to commit misconduct is complex. Arturo Casadevall of the Albert Einstein College of Medicine in New York and editor in chief of the journal mBio, places a large part of the blame on the economics of science. “What is happening in recent years is that the rewards have become too high, for example, for publishing in certain journals. Just like we see the problem in sports that, if you compete and you get a reward, it translates into everything from money and endorsements and things like that. People begin to take risks because the rewards are disproportionate.”

As a PhD student in the 1980s, Casadevall says he published research in a few different journals depending on what his research was about. “Within 10 years, all you heard was, ‘Where is the paper going to be published?’ not ‘What’s in it?’. Scientists have got into this idea that where you publish determines the value of the work and that’s crazy. What’s important is what’s in the paper.”

Casadevall and Fang are aware that their spotlight on misconduct has the potential to show up scientists in a disproportionately bad light – as yet another public institution that cannot be trusted beyond its own self-interest. But they say staying quiet about the issue is not an option.

“Science has the potential to address some of the most important problems in society and for that to happen, scientists have to be trusted by society and they have to be able to trust each others’ work,” Fang says. “If we are seen as just another special interest group that are doing whatever it takes to advance our careers and that the work is not necessarily reliable, it’s tremendously damaging for all of society because we need to be able to rely on science.”

For Simonsohn, the biggest issue with outright fraud is not that the bad scientist gets caught but the corrupting effect the work can have on the scientific literature. To reduce the potential negative effects dramatically, Simonsohn suggests requiring scientists to post their data online. “That’s very minimal cost and it has many benefits beyond reduction of fraud. It allows other people to learn things from your data which you were not able to learn about, it allows calibration of other models, it allows people to, three years later, reanalyse your data with new techniques.”

Ivan Oransky, editor of the Retraction Watch blog that collects examples of retracted papers, argues: “The reason the public stops trusting institutions is when [its members] say things like, ‘There’s nothing to see here, let us handle it,’ and then they find out about something bad that happened that nobody handled. That’s when mistrust builds.The big challenges that face humanity, says Casadevall, are scientific ones – climate change, a new pandemic, the fact that most of our calories are coming from a very few plants, which are susceptible to new pests. “These are the big problems and humanity’s defence against them is science. We need to make the enterprise work better.”

Malpractice and misconduct

The South Korean scientist Hwang Woo-suk, rose to international acclaim in 2004 when he announced, in the journal Science, that he had extracted stem cells from cloned human embryos. The following year, Hwang published results showing he had made stem cell lines from the skin of patients – a technique that could help create personalised cures for people with degenerative diseases. By 2006, however, Hwang’s career was in tatters when it emerged that he had fabricated material for his research papers. Seoul National University sacked him and, after an investigation in 2009, he was convicted of embezzling research funds.

Around the same time, a Norwegian researcher, Jon Sudbø, admitted to fabricating and falsifying data. Over many years of malpractice, he perpetrated one of the biggest scientific frauds ever carried out by a single researcher – the fabrication of an entire 900-patient study, which was published in the Lancet in 2005.
Advertisement

Marc Hauser, a psychologist at Harvard University whose research interests included the evolution of morality and cognition in non-human primates, resigned in August 2011 after a three-year investigation by his institution found he was responsible for eight counts of scientific misconduct. The alarm was raised by some of his students, who disagreed with Hauser’s interpretations of experiments that involved the, somewhat subjective, procedure of working out a monkey’s thoughts based on its response to some sight or sound.

Hauser last week admitted to making “mistakes” that led to the findings of research misconduct. “I let important details get away from my control, and as head of the lab, I take responsibility for all errors made within the lab, whether or not I was directly involved,” says Hauser in a statement sent to Nature. The doubts over Hauser’s work affect a whole field of scientific work that uses the same research technique.

This article was amended on 14 September 2012. The original referred to Liz Wager of the Committee on Public Ethics rather than Publication Ethics. This has been corrected. This article was further amended on 19 September 2012. The original stated that a 2006 analysis of the images published in the Journal of Cell Biology found that about 1% had been deliberately falsified. This has been corrected.