When Your Stats Are Exploratory, Say So Up Front
A reviewer flagged my entire results paragraph. Not one number was wrong -- nothing in the writing said which analyses were planned and which were fishing, so he had to assume the worst.
Show a reviewer a paragraph of statistics without labels, and most of them read as confirmatory. That's the default. In an abstract that mixes field observation with a sealed follow-up, it's also almost always wrong.
I hit this last week revising a conference abstract. The field-data paragraph had three deployments, half a dozen statistics per deployment, a couple of them impressive. A domain reviewer flagged the whole paragraph as the single biggest inferential vulnerability in the document. Not because any number was wrong. Because none of them were labeled, and there was no dated document from before data collection saying "we will run runs test + Markov + change-point + spectral, corrected at α = 0.05 across the family." So he assumed, honestly and by field convention, that I picked those tests after I saw the data.
That's the garden of forking paths, with ordinary multiplicity sitting on top of it. And it's a defense you get to make once, if at all.
Prespecified vs. exploratory
Prespecified means the estimand, the test, the threshold, and the multiplicity correction were all decided before anyone looked at the data, and written down somewhere dated. Exploratory means the statistic was picked or tuned after the data existed. Running five tests and reporting the one that hit is exploratory. So is choosing a window size after inspecting the series, or noticing a pattern and computing the thing that captures it.
An exploratory p-value doesn't mean what a confirmatory one means, because you're implicitly testing a whole family. Take five roughly independent tests: the chance at least one clears p = 0.02 by luck alone is about ten percent -- 1 − 0.985. That's the floor, and it only holds if those five were the whole search. Tune the tests, pick the window after seeing the series, or leave a few tried-and-dropped analyses unreported, and the real rate climbs above it.
prespecified exploratory
chosen before the data exist after seeing the data
written down dated plan: test, nowhere -- it lives in
threshold, correction the analysis session
p = 0.02 means what it says one draw from a family
you searched
belongs in the confirmatory claim the motivation for one
None of this makes exploratory analysis wrong. Field observation is exploratory by design, and it should be -- you can't preregister a natural experiment you didn't know was coming. But you can't drop an exploratory p-value into the same slot as a confirmatory one and let the reader guess which is which. They'll read it as confirmatory. Then they'll mark down everything else in the document to compensate.
The framing fix
One sentence, near the top of the paragraph where the prior analyses start:
These are exploratory observations from field deployments; the sealed follow-up runs the prespecified confirmatory test.
Adapt the wording. What it does is the point: it labels the prior stats as exploratory, positions the follow-up as the actual confirmatory work, and shows you know which side of the line each analysis sits on. That last part is what a careful reviewer is really checking for.
Be clear about what the sentence buys you, though. It fixes the claim, not the inference underneath it. The exploratory numbers stay exploratory. What makes the eventual claim confirmatory is the follow-up: a precommitted analysis, fixed before the confirmatory data exist, run on fresh data. Discover in one dataset, confirm in another. Labeling just stops you from spending credibility you haven't earned yet. And often the abstracts that get flagged here aren't flagged because the analysis is bad -- they're flagged because nothing in the writing shows the author knows that difference.
What it looked like for me
The concrete case: a psi-adjacent field-observation platform -- the Ionosphere chandelier, hardware QRNG, audiovisual feedback, three deployments over several months. Aggregate stats null across all three. Fine. Then the exploratory battery -- runs test, Markov transition probabilities, cluster counts, change-point, spectral -- hit on the attended unit and stayed null on the paired control. Numbers a reader looks at twice.
That battery wasn't preregistered. We ran the platform, collected several thousand 60-second periods, and picked interesting-looking statistics to describe what we'd found. Standard exploratory work. The reviewer read the numbers as confirmatory claims they can't support. One clause -- labeling the paragraph as exploratory motivation, and reserving confirmatory language for the sealed follow-up, where the analysis plan is locked before the data exist -- flipped the read from "some of these look shaky" to "this is what motivated the sealed test." Same numbers, same finding, entirely different credibility position.
When the abstract is tight
There's a compressed version. Instead of "we found X (p = 0.02)," write "an exploratory analysis showed X (p = 0.02, uncorrected)." Two words and a parenthetical, same job, and it survives compression to almost any word cap.
Reviewers can't tell prespecified from exploratory unless you tell them, and left guessing, they assume confirmatory. An uncorrected exploratory p-value in the confirmatory slot reads as overclaiming even when the work behind it is careful. One sentence, in the right place, is what stops it.