Why a forward-only arena cannot backtest

The first arm buys filings before EDGAR has them. On 47 of the 64 filings that carry an acceptance receipt it entered a mean of 6.05 hours before acceptance, and at most 7.88 hours before, so no account could have placed those trades. Switching to the clock that was reachable takes the same 88 filings from +5.13% to −2.02%, and the gap between the naive arm and the receipt arm is 4.95 points, recorded in the file as lookahead_inflation.

The third row rests on 17 filings. pit and dump enter on different sessions for only 17 of the 88, so only those 17 can differ at all. Across them the per-filing difference averages +0.95% with a standard deviation of 2.92%, and 9 of the 17 go the pit arm’s way. That is roughly 1.3 standard errors from zero, so the run sizes the effect and leaves its sign open. The 2.20 points describes this basket in this window and nothing wider. The basket rule, the exclusions and the per-event table are on methods.

A forward-only board cannot run the table above, because the window is in the past and the board’s window starts at its own launch. The nearest thing it can do is wait for 88 comparable events to arrive. Our five contiguous months produced exactly that count across a twelve-issuer basket, so the same sample costs roughly five months of calendar, plus the time to notice that the sample is still small.

The gold-winning entrants in CLEF 2026 task 3 report that one submission’s return moved from +2.0% to +13.51% between two snapshots of the same live window, with rank moving between 1st and 13th, and their own conclusion is that “live agent rankings are unstable”. Replay returns the 88-event result the same day, and returns it again on a re-run. The forward route costs about five months of calendar and ends at a ranking its own authors call provisional.

A live window happens once, so a reader cannot re-run it, and model stochasticity moves the number even for the operator running it twice (AlphaForgeBench, arXiv 2602.18481). Of 19 primary LLM-trading studies surveyed in arXiv 2605.19337, two report time-consistent splits and none reaches reproducibility. The three contamination controls are compared side by side on the other page.

Step 6 is what produced the 47-of-64 figure above. The measurement is three lines over rows you already have: take the entry timestamp your join implied, subtract acceptance_at, and count the filings where the difference is positive. Any backtest over EDGAR can run that check against its own trades, and a vendor without the receipt clock cannot.

Cutting the retrieval side bounds what reaches the model. It does nothing about the model having read the period during training, and with no filing text on our rows yet, that memory is carrying most of the narrative. Our four-arm run over SVB week measured it: given only the string SVB FINANCIAL GROUP 8-K, the model produced a dollar figure for the capital raise and a description of the securities sale, neither of which appeared in the payload.

The control arm is how you size that channel. Withhold the issuer name and move every date forward ten years, then run the same prompt. In our run the same model returned a neutral view on every day of that week, receivership day included, which puts the signal in the weights. If your numbers survive the shift, the payload was doing the work. This is also the one place a post-cutoff forward window genuinely beats replay, and the two controls compose: use replay for the history and a post-cutoff window for the weights.

• The certified backfill ends 2023-03-31 and the tape restarts 2026-08-26, so 2024 and 2025 cannot be replayed here.
• 64 of the 88 filings carry an acceptance receipt and 24 fall back to the day-end bound, so the pit curve is partly the dump curve until the backfill stamps more rows.
• SIVB and WBA are excluded because the price source returns no bars for them, which removes SVB’s own filings from all three curves.
• Returns come from one basket over one five-month window with a five-session hold, so they size a timing error over that sample.
• PIT rows carry filing index metadata rather than filing text, so fetch the document from source_locator when the run depends on what a filing says.

$ curl -s "$PIT/v1/sample/news?example=svb&as_of=2023-03-09T20:00:00Z"
$ curl -s "$PIT/v1/sample/news?example=svb&as_of=2023-03-10T23:59:59Z"
$ curl -s "$PIT/v1/sample/coverage?example=svb"

The first answers count 0 with both sources certified complete for 9 March. The second returns the receivership 8-K, accession 0001193125-23-067777, whose acceptance_at is 2023-03-10T22:23:03Z. The same instant asked with visible_by=available_at answers 409, because that clock is null on SEC rows and we do not fill it in from a receipt.