Bad statistics can get your paper rejected

Three brainy researchers from Leiden and Amsterdam checked 513 papers from top journals, looking for something specific: a paper that hit the statistical significance mark and another that did not. The study covered 157 of those papers. In 79 of them, the authors reasoned from the two separate tests, and in 78 they tested the difference directly. According to reviewers who are also scientists, the right and wrong ways were about 50/50 in the top five journals.

These statistical mistakes are common in papers and happen to many researchers. Their resemblance to conventional analyses, which they derive from foundational texts studied during postgraduate education, allows them to survive. These similarities make it hard for authors who have been looking at the same results table for a while to notice them.

Proficient statistical reviewers will examine your results, checking the numbers against each other instead of your conclusions, as some reviewers are familiar with the list and take notes. These numbers are from psychology, neuroscience, and biomedicine because they have studied themselves, but the trends are not just for fields that do certain tests on animals.

Which statistical errors show up most often?

Eight patterns account for most of what a statistical reviewer flags, and all of them are detectable from the manuscript alone, without your raw data.

How do reviewers find errors without your input?

The analysis starts by considering degrees of freedom, as this is the only place in the results that shows the sample’s relevance.

For the audit, you need only a printer to place your results section next to the methods page that gives your N. For every t, F, and χ² in the paper, read the degrees of freedom against that N and check that the arithmetic works. The t-test comparing 30 animals to another 30 animals showed t(58). Stanley Lazic found that 12% of the papers in a Nature Neuroscience issue had fake replication, and 36% could not be reproduced because of insufficient details.

Similar audits can run for subtle issues; a t(58) should be reported for a two-sample comparison of two groups involving 60 participants. Sometimes you can have a good reason, but you forgot to give it. A reviewer has no way to figure out the reason they would not accept, so they ask.

Michèle Nuijten and her team created Statcheck, an R package. This software goes beyond manual audits, extracting null-hypothesis test results from papers and recalculating p-values using their original test statistics and degrees of freedom. They found that in studies from eight psychology journals (1985-2013) with over 250,000 p-values analyzed, 50% had p-values that did not align with their stated statistics. One out of every eight papers had an inconsistency substantial enough to alter a result’s significance status relative to the .05 level. The web version takes a PDF and returns a list. Nicholas Brown and James Heathers’ GRIM test does the equivalent job for means of integer data: given a sample size and a scale, only certain means are arithmetically possible. Of 71 articles they could test in leading psychology journals, around half contained at least one impossible mean.

Both tools are incapable of spotting misconduct or wrongly flagging faulty data. Check the list, fix errors, and add any missing items or the precise p-value if the tool could not find it.

Why should you not compare two significance levels?

Basically, if significance relies on the specific test, saying “significant here, not significant there” implies you ran two tests, and they are not the ones you are discussing.

The first estimate, 25 with a standard error of 12, exceeds the .05 threshold compared to the second, which is 22 with a standard error of 13. Their difference is 3, with no significance when compared to its standard error. The paper says there was an effect in older adults, but not younger ones, after describing two tests and making a claim about a third test that did not actually happen. It is possible that the statement is accurate, yet the proof is lacking.

Fix this by checking whether the model has an interaction term, a contrast, or a direct comparison of the two estimates. The statistics should be placed in the results instead of the pair of separate p-values, and add the means when the interaction misses significance while one simple effect clears it. Nieuwenhuis and colleagues note that the error is much more common in cellular and molecular work than in the systems and cognitive papers they looked at.

How does p-hacking appear to someone reviewing it?

It seems like they only reported the good statistics.

Nobody sees you drop the two participants who ruled the effect out, or try three transformations of the outcome before finding the one that worked. Joseph Simmons, Leif Nelson, and Uri Simonsohn showed with experiments that a handful of these ordinary flexibilities, used in combination and never disclosed, are enough to push the false-positive rate far above the stated 5%. Megan Head and colleagues then text-mined p-values across disciplines and found the fingerprint at scale: an excess of reported values just below .05. Megan Head and her colleagues reached a measured conclusion. They saw that the bias showed weakly against actual effect sizes in meta-analyses. However, they confirmed the extreme exists and is not an artifact of how science functions.

Reviewers notice things in the results that were not mentioned in the methods. A sample size with no reason and a suspiciously round link to the significance threshold. In the report, they removed participants who did not follow directions, but did not provide the numbers. Subgroups that show up for the first time in the abstract’s headline claim. Any of these can be innocent, and reviewers treat several of them together as a pattern.

The defense is disclosure, applied at the point of writing. You can describe how you determined the sample size prior to the dataset, or state that you did not predetermine it. Report every result you measured, including the ones that went nowhere, even if they live in a supplementary table. Give exclusions as counts with reasons, grouped by group, so the numbers add up to the N you started with. To avoid reviewer criticism, classify data-driven decisions as exploratory and validate confirmed findings with planned analyses in a short methods section.

When does an analysis go in circles?

When a pick rule and a statistical test use the same data.

Nikolaus Kriegeskorte and his team called this “double dipping” in their fMRI research. Basically, they choose the brain areas that react most strongly to something, and then they check if that thing is actually there. The problem is, this method can make it look like something is there even if it is just random noise. Since the structure is so widespread, the name did not change. Find the cells that react and then write how strongly they react. Categorize subjects based on their scores and then check if those categories differ on the same measure. Filter outliers based on the dependent variable, select a time frame, and then evaluate the difference within that frame.

When a few outliers can skew the whole thing, Tamar Makin and Jean-Jacques Orban de Xivry, who wrote the review for eLife, show the same problem of making connections that do not really mean anything. Waste an hour on the examples before your next r report.

Make sure your draft follows all the rules. Also, track where your data comes from. If you test on the same data you based your analysis on, it is a bit of a dead end. In such cases, the repair involves independent selection, meaning the use of a separate dataset, held-out data, cross-validation, or a pre-established and clearly stated rule.

What do reviewers notice in your figures that they miss in your text?

They look at how the observations spread, which a sentence reporting a mean and a p-value cannot show them.

Tracey Weissgerber and her team analyzed 703 articles from the best physiology journals. They noticed that most researchers use data in bar and line charts, while they rarely use scatterplots, box plots, and histograms. That matters because many distributions collapse to the same bar. A clean separation between two groups and a pair of overlapping clouds pulled apart by one extreme value draw identically once you reduce them to a mean and an error bar.

When reviewers see a bar chart with seven observations per group, they know it is an inkblot. Plot the points. For smaller groups (less than 20), a scatterplot with the mean works. It will show everything the bar chart did, how spread out the data is, and the actual count. You can also visually audit the first section and count the points against the sample size in your methods.

What’s the best way to write results so a statistician trusts them?

Provide sufficient detail for verification of calculations, a task simpler than it appears and often omitted in publications.

You should add minimum information, such as the test and software name, N for groups and cells, test statistic with df, exact p-values, and relevant confidence intervals. SAMPL guidelines outline biomedical research rules, and the EQUATOR Network offers reporting guidelines for various study types. Lazic said that if you report your sample size, the statistics, and exact p-values, then anyone can spot mistakes like his.

Their trust is won or lost based on how you interpret things. The 2016 report from the American Statistical Association outlined what a p-value does not show, along with interpretations by Sander Greenland and colleagues that appear in published works, largely concerning the use of .05 as a dividing line for significance. Without significant results, we cannot prove the lack of an effect, and confidence intervals that include clinically meaningful effects do not support a claim of “no difference.” In 2019, over 800 researchers added their names to a comment published in Nature arguing for discontinuing the classification entirely. Reviewers are careful, even if journals are slow, and a “trend toward significance” will now get a note in the report rather than being waved through.

None of this can make the analysis correct. This makes analysis easy to read, and this is one where you can spot your own mistakes just as easily as the next person can.

References and useful policies

1. Nieuwenhuis S, Forstmann BU, Wagenmakers EJ. Erroneous analyses of interactions in neuroscience: a problem of significance. Nature Neuroscience. 2011;14(9):1105-1107.
2. Lazic SE. The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?. BMC Neuroscience. 2010;11:5.
3. Nuijten MB, Hartgerink CHJ, van Assen MALM, et al. The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods. 2016;48(4):1205-1226.
4. Nuijten MB, Epskamp S. statcheck (web application).
5. Brown NJL, Heathers JAJ. The GRIM test: a simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science. 2017;8(4):363-369.
6. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. 2011;22(11):1359-1366.
7. Head ML, Holman L, Lanfear R, et al. The extent and consequences of p-hacking in science. PLOS Biology. 2015;13(3):e1002106.
8. Kriegeskorte N, Simmons WK, Bellgowan PSF, et al. Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience. 2009;12(5):535-540.
9. Makin TR, Orban de Xivry JJ. Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. eLife. 2019;8:e48175.
10. Button KS, Ioannidis JPA, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14(5):365-376.
11. Weissgerber TL, Milic NM, Winham SJ, et al. Beyond bar and line graphs: time for a new data presentation paradigm. PLOS Biology. 2015;13(4):e1002128.
12. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. The American Statistician. 2016;70(2):129-133.
13. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350.
14. Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305-307.
15. EQUATOR Network. SAMPL guidelines: basic statistical reporting for articles published in biomedical journals.