Compare

Three ways to control contamination, and what each one can replay

Any benchmark built on public text has to answer one question: could the model already know the answer? There are three working answers. Hold the questions privately, so nothing published could carry them. Score only on events dated after the model’s training cutoff. Or keep the record with the clocks that say when each row became knowable, and read it back at a chosen instant. Two of the three can only score events that have not happened yet. Below is what each one measures, how long it takes, and who can check the result.

Updated 2026-08-26. PIT figures read off our own measured run; outside figures carry their source in the row.

Evidence

One live board, two readings of the same submission

The gold-winning entrants in CLEF 2026 task 3 published the instability of their own ranking. One system’s reported return moved between snapshots of the same live window, and rank positions moved with it.

CLEF 2026 task 3 — one submission, two leaderboard snapshots - earlier snapshot return +2.0% + later snapshot return +13.51% rank on the same board moved between 1st and 13th

Their own conclusion is that “live agent rankings are unstable” and that any single snapshot should be read as provisional. CLEF task 3 has real entrants and a published self-assessment, which makes it the best-documented of the live arenas. These figures describe the method at its most careful.

Methods

Secrecy, freshness, provenance

Each column stops the contamination in a different place.

	Secrecy	Freshness	Provenance
The control	The questions are held privately, or they have not happened yet. A live arena is the strong form: the events arrive after the run starts, so no model could have trained on them.	Score only on events dated after the model’s published training cutoff.	Keep the timestamps and hashes that say when each row became knowable, then bound every read to a chosen instant.
Replay a past window	No. A set that has never been published has no past to read back.	No. The window is defined against a cutoff, so it starts at the cutoff.	Yes. The instant is a query parameter, so 2022-11-14 is as reachable as today.
Time to a result	As long as the events take to happen.	The same wait, plus the delay before enough post-cutoff events exist.	The time the query takes.
Shelf life	One use per question. Publishing the answers ends the control.	Ends when the next model ships with a later cutoff and the window falls inside it.	None. A stamped row stays stamped, and re-reading it does not consume it.
Who can check the run	The operator. A private set cannot be re-run by a reader.	Anyone who takes the published cutoff dates at face value.	Anyone holding the same corpus and the same instant. The rows come back identical and the partitions carry their own SHA-256.
What it does not touch	What the model remembers about the wider world going into the round.	Nothing about retrieval. A fresh window can still be joined against a file downloaded today.	Replay bounds the rows that reach the model at query time. Its training memory of the period stays intact.

Construction

Why the first two only run forward

Three properties, and they hold however long a board has been running.

Property	What follows from it	Source
No history	A question that did not exist in 2022 cannot be asked about 2022. Both controls get their strength from the answer being unavailable at training time, and a past window fails that test by definition.	Definitional.
Months per sample	Sample size accrues at the rate the world produces events. Our own five contiguous SEC months, 2022-11-01 to 2023-03-31, yielded 88 tradable 8-K events across a twelve-issuer basket. A forward-only board needs about that much calendar to reach the same count.	Measured on our corpus; the per-event table is on methods.
Unreplicable windows	A live window happens once. A second reader cannot re-run it, and model stochasticity moves the result even for the operator running it twice.	AlphaForgeBench, arXiv 2602.18481, on live evaluation not being independently replicable.

The literature record matches. Of 19 primary LLM-trading studies surveyed in arXiv 2605.19337, two report time-consistent splits and none reaches reproducibility. A separate review of 164 papers (arXiv 2602.14233) found lookahead bias addressed in 26.8% of them, with 74% of surveyed practitioners calling evaluation tooling scarce or absent.

Before the first round

What a forward-only board shows while it waits

A board that scores forward has nothing to display until its rounds run. DataCedar publishes an Arena methodology, last revised 26 Jul 2026, with SHA-256 packet digests, a known_at ≤ t₀ rule and six frontier models listed. Read on 2026-08-26, every cell at datacedar.com/arena carried “Awaiting first eligible round” and “No certified execution ranking yet”, with the counters at 0.

Every entrant that started at the leaderboard stage is waiting on the same calendar. StockBench claimed a post-cutoff window in October 2025 and that window now sits inside the training data of 2026 models, LiveTradeBench has stopped, and Alpha Arena has not run a round since December 2025.

The alternative check

Detecting contamination after the fact

A fourth option is to skip the control and test the finished model for memorization statistically. The most recent measurement of that approach reports 201 correct outcomes across 335 evaluations, which is 60%, and its authors state that statistical detection “cannot yet replace transparent data provenance” (arXiv 2606.03305). At 60% it works as a screen over a batch of models. Carrying a single published result takes a control at the record.

The 2022 question

The one question only provenance answers

“What would this agent have done in March 2023?” needs a record that still knows what was knowable then. Two calls, one instant apart.

the same query either side of one 8-K 0 → 1

$ curl -s "$PIT/v1/sample/news?example=svb&as_of=2023-03-09T20:00:00Z"
$ curl -s "$PIT/v1/sample/news?example=svb&as_of=2023-03-10T23:59:59Z"

The first answers HTTP 200 with count 0 and both sources certified complete for 9 March, so the zero says no filing existed at that instant. The second returns the receivership 8-K, accession 0001193125-23-067777. Run either one again tomorrow and the same rows come back, because the cut is a function of the record and the instant.

The row also carries acceptance_at at 2023-03-10T22:23:03Z from its EDGAR receipt, which is what lets you measure how early a day-resolution join would have traded. Coverage for every day behind the cut travels in the same envelope, so an empty answer and an unread day come back as different HTTP status codes. The clock rules are on clocks and the certificate is defined on coverage certificate.

Limits

Where forward-only is the better control

A window dated after every published cutoff is the only one of the three that touches the weights. Replay bounds what reaches the model and leaves its memory of the period intact.
Our certified backfill ends 2023-03-31 and the tape restarts 2026-08-26, so 2024 and 2025 are outside what you can replay here.
Our own forward window began recording on 2026-08-26, which makes it young. A board that started in 2025 holds more forward history than we do.
SEC publishes no dissemination clock, so a strict visible_by=available_at cut over SEC rows answers 409. We hold no evidence of when EDGAR released the document.
PIT rows carry filing index metadata and not filing text yet, so a model reading a cut supplies the narrative from its own memory. Fetch the document from source_locator when the run depends on what a filing says.

Check it

Run the 2022 question yourself

The playground takes no key and watermarks every response sample: true, so you can see a cut refuse and a cut answer before deciding anything.

Get access Run the moment

The practitioner walkthrough with the measured arms is on why forward-only cannot backtest.