Lisa Clancy

In 2005, John Ioannidis, a Professor of Medicine at Harvard University, published a paper titled “Why most published research findings are false,”¹ which convincingly demonstrated that “for most study designs and settings, it is more likely for a research claim to be false than true.” In 2012, Begley and Ellis showed that 47 of 53 “landmark” cancer studies could not be reproduced.² These papers, among others, brought widespread attention to the reproducibility crisis within the scientific community and significantly advanced the field of meta-research, which focuses on studying and improving scientific practices.

A considerable body of evidence suggests that questionable research practices (QRPs), employed by a surprisingly large number of academics, may be contributing to the high rate of Type 1 errors in research articles. Type 1 errors are false positives or findings that are not true effects. Here, we discuss some common QRPs³:

• Data peeking: Data collection is stopped earlier than planned upon finding results that align with preconceived expectations, thereby introducing bias into the study outcomes.
• Excluding data based on results: Decisions about excluding data are made after observing the impact of such exclusions on the results, potentially skewing the findings.
• Failure to report all conditions: A complete account of the experimental or situational conditions influencing the outcomes is omitted, which can provide an incomplete or misleading interpretation of the factors contributing to the results.
• HARKing (Hypothesizing After the Results are Known): Hypotheses are developed or adjusted retrospectively after the results have already been analyzed, creating a false impression that the hypotheses were pre-specified.
• Hiding demographic moderators: Claims are made that results are unaffected by demographic variables (e.g., gender, age) despite uncertainty about, or knowledge of, their actual impact, resulting in incomplete or biased reporting.
• Opportunistic rounding of p-values: Statistical significance is misrepresented, such as by reporting a p-value of 0.054 as being less than 0.05, thereby exaggerating the strength of the findings.
• Optional stopping: Additional data collection is undertaken after an initial analysis reveals non-significant results, which can artificially increase the likelihood of finding significant outcomes.
• Selective reporting of outcomes: Only results that support the hypotheses are highlighted, while those that do not align are omitted, creating a distorted representation of the evidence.
• Selective reporting of performed analyses: Analyses that yield results supporting the predictions are reported exclusively, making it appear that there is robust evidence for the theory, while non-supporting analyses are ignored.
• Selective reporting of what “worked”: Findings that align with the hypothesis are emphasized, while contradictory or non-supportive results are downplayed or excluded. This creates a misleading narrative of consistent support for the hypothesis.
• Selective inclusion of covariates: Covariates that fail to achieve statistical significance (e.g., p > 0.05) are omitted from the report, resulting in an incomplete depiction of the data and oversimplified relationships.
• Selective switching of analysis methods: Analytical methods are changed midway through the study to obtain more favorable results, increasing the likelihood of false positives and undermining the reliability of the findings.

The findings from a recent survey involving 6,813 participants shed light on the prevalence of research misconduct and QRPs within the scientific community. The data revealed that 4.3% of respondents admitted to fabricating data, while 4.2% reported instances of falsification. Moreover, the survey uncovered that half of the researchers involved engaged frequently in at least one form of QRP over the past three years.⁴ These results highlight the need for heightened awareness and stronger preventive measures to ensure the integrity and reliability of scientific research.

Sources:

1. Ioannidis JP. Why most published research findings are false. PLoS Medicine. 2005 Aug 30;2(8):e124.
2. Begley CG, Ellis LM. Raise standards for preclinical cancer research. Nature. 2012 Mar 29;483(7391):531-3.
3. Lakens D, Mesquida C, Rasti S, Ditroilo M. The benefits of preregistration and Registered Reports. Evidence-Based Toxicology. 2024 Dec 31;2(1):2376046.
4. Gopalakrishna G, Ter Riet G, Vink G, Stoop I, Wicherts JM, Bouter LM. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands. PloS One. 2022 Feb 16;17(2):e0263023.