For eval builders
It comes up in every eval review, and over public data it usually has no answer. The model trained on what happened next, so a benchmark built from 2021 filings scores recall as much as reasoning. PIT answers it with three products that fit together: an open-source harness that runs your agent with the network off, benchmark releases that ship their own control arms, and a signed certificate anyone can check without trusting either of us.
The playground takes no key and watermarks every response
sample: true, so you can run a cut before you decide.
Pick a moment
What you use
The first is free and runs on your laptop, the second is what it reads, and the third is what you hand to the reviewer who asked the question.
01
Your agent, a benchmark window, network off. Every tool call is intercepted and served only from that window, and the whole trajectory is hashed into a run receipt. It is Apache-2.0, needs Python 3.11 or newer, and has no runtime dependencies.
pip install -e harness/
02
A frozen window with matched control arms: the rows as served, the same rows with the issuer redacted, and the same rows with every date moved ten years forward. The score that survives the shifted arm came from the rows rather than from the model’s memory.
03
A signed verdict on one run: what the agent was shown, when each
item was knowable, the control-arm deltas, and the
reasoning-audit findings. Anyone holding it can check it against
the corpus with certverify, which costs nothing and
needs no account.
What you run against
Both windows come out of the same corpus and answer different questions, so most harnesses end up using each for what it is good at.
2022-11-01 → 2023-03-31
An offline release of five contiguous SEC months, named
pit-eval-sec-2022-11_2023-03. Three arms ship over
the same 287,929 rows: the rows as served, the same rows with the
issuer removed, and the same rows with every date moved ten years
forward. Score all three, and the score that survives the shifted
arm is the part your model read rather than remembered.
MANIFEST.json carries a SHA-256 for every file and
pins the corpus version. receipts.jsonl carries one
line per calendar day, holding that day’s coverage
certificate. The release comes with a paid plan rather than as a
separate license, and we hand it over by email while the
text-complete edition finishes:
operations@aqx.llc.
2026-08-26 → the last tick
A job has been writing the public record into the corpus every fifteen minutes since 2026-08-26, stamping each row with the minute we first saw it. Every week it runs adds eval material dated after any published cutoff, so the calendar does the work that redaction would otherwise have to do.
A lane says how a row got its timestamp. Tape rows carry
forward_first_seen, and a response never mixes them
with the backfilled certified_pit rows. What we
observed therefore stays separable from what the publisher
asserted.
What the control arms caught
Four arms over identical PIT payloads at temperature zero, walked day
by day through SVB’s receivership week, which is the most
memorized week in recent filings history. Arm A saw exactly what
/v1/news returns and flipped to distress on
8 March, two days before receivership, citing an urgent
$2.25B capital raise. The only new row that day was
the string SVB FINANCIAL GROUP 8-K. PIT returned no
amount, no losses and no urgency, so the model supplied all three from
its memory of March 2023. Arm D held the same rows with the
issuer withheld and every date moved to 2033, and it answered
neutral/0.5 on every date, receivership day included.
A person caught that by reading the trace. The reasoning audit is that check as a product: it pulls every number, entity, date and event assertion out of the agent’s rationale and matches each one against the rows the run actually served, then quotes and scores the ones that appear nowhere. A certificate carries those findings alongside the arm deltas, which is what makes the score defensible rather than merely reported.
The release ships arms C and D as files, so the comparison runs on your own harness. Run the moment yourself
Getting a window
Install the harness and it runs today against the public sample, with no account and nothing to cancel. A subscription is what buys the record it reads: Power reads every row we hold, back to 2008-09-01, and under those rows sit 415 parquet partitions, one file per source per day, 178 MiB in total, each carrying its own SHA-256. A harness that reads files finishes a long walk faster than one paging the API, and both read the same rows. Researcher at $29 covers the same corpus at 60 requests a minute if you are still deciding.
Self-serve covers evals you run for yourself or your employer. If the benchmark ships PIT rows to your own users, that is a commercial license. The harness itself is Apache-2.0 and needs no license from us.