Lock One Analysis Battery Across All Datasets
Six datasets, five analyses, each choice defensible on its own -- and together, indefensible. What happened when I ran one fixed battery over every dataset instead.
Six datasets, five analyses. That was my corpus this morning. The first experiment got event-window comparisons because there was an event worth windowing. The second got a runs test and a Markov chain because the data looked clustered. The third got spectral analysis because someone saw a rhythm. Each choice was defensible on its own. Together they're indefensible, and a reviewer will say so in one sentence: you ran different tests on each dataset until something hit.
The fix
The fix took an afternoon. Write down one battery -- every statistic, every threshold, every parameter -- and run it identically over every dataset you have. Mine ended up as:
- aggregate stats against the theoretical rate
- burst rate per hour
- a runs test on the threshold-crossing indicator
- a top-n exceedance sweep
- a CUSUM change-point with a seeded permutation p
- Fisher's g for periodicity
Threshold 1.96, sweep to n=10, 2000 permutations, seed written into the script. Eleven units through the same pipe, one table out.
Two things I didn't expect
First, the results I was most attached to survived. The dataset with the dramatic clustering signal still shows it under uniform treatment -- same runs test, same parameters as everything else, still off the charts. The practitioner session's tail excess reproduced to the third decimal. I had been half-braced to lose something, because bespoke analyses have a way of flattering their datasets. When a result survives the transition from custom analysis to locked battery, you learn it wasn't an artifact of test selection -- not that it's real, since a mis-specified null or a lurking confound can still fool you, but one whole class of doubt is gone. When it doesn't, better to know now than in review.
Second, and more useful: the battery broke a story I was starting to tell. I had a tidy narrative forming -- the more deliberately someone engages the apparatus, the more tail structure shows up. Ordered by engagement, the top of the table cooperated beautifully. Then the dataset with the single most famous event in the whole corpus landed dead last, weakest tail of the eleven. Because its signal was always local to a twenty-minute window, and the battery scores whole datasets. Diluted across three thousand periods, the event disappears. The tidy narrative needed an asterisk the size of the table: global statistics and event-local statistics are different claims, and a scaling figure built on whole-dataset units would have been quietly wrong in a way nobody -- including me -- would have caught by eyeballing per-dataset writeups.
whole-dataset score [========|===============] ~3000 periods, one number
^ the 20-minute event, in here somewhere
-> ranked last of eleven on tail excess
event-window score [|] the same twenty minutes, scored alone
-> the result the corpus is known for
That's the actual payoff. Not tidiness. A locked battery is a machine for noticing which of your findings are commensurable and which aren't. Custom analyses hide category errors because every table has different columns; you can't see that two numbers shouldn't be compared when they never appear side by side.
Practical notes
Put the seed and the permutation count in the script, not your head. If one dataset has a different sampling rate, run the battery anyway and mark the unit -- excluding it is a bigger researcher degree of freedom than annotating it. Keep the battery dumb: the moment you add a statistic because one dataset seems to want it, you've rebuilt the problem you were solving. Event-window and other dataset-specific analyses still happen -- they just happen in a second layer, clearly labeled, on top of a base table where every cell means the same thing.
One thing the uniform treatment doesn't buy you: validity. If the series are serially dependent -- and burst-y time series usually are -- a plain permutation null for the change-point p can be invalid, because permuting scrambles the very time structure the test is supposed to be reading, so the p ends up measured against the wrong null. Same battery, same flaw on every dataset. The battery makes that flaw consistent and visible instead of hiding it behind eleven different procedures; it doesn't make it go away. Comparability is not the same as being right, and a locked battery buys the first, not the second.
The battery doesn't make your analyses good. It makes them comparable, which is the precondition for anyone -- reviewer, collaborator, you in six months -- telling whether they're good.