The Long-Horizon Register

Across 995 in-charter evaluation artifacts — every one of them — 619, or 62.2%, never pin down the construct they are built to measure. Not buried in an appendix: absent. We checked bodies precisely because we expected the definitions to be hiding there.

The absence has a characteristic shape. Papers reach for "long-horizon" and "long-term" as motivating vocabulary in the introduction, then define difficulty by some other quantity entirely — or by nothing. Four of the most-cited agent benchmarks in the corpus illustrate it:

This is not a complaint about sloppy writing. It has a measurable consequence, which is the subject of the next section: when the term goes undefined, the quantity it stands for drifts across seven orders of magnitude without anyone in the field having to notice.

Every benchmark that stated the length of a single example was recorded in its own units, with no conversion. Two ladders below plot those stated lengths on log axes — one for discrete units (steps, turns, sessions, tasks), one for wall-clock durations. Each mark is one benchmark's stated example length. No benchmark appears on both ladders unless it reported both.

The two ladders are not commensurable with each other, and that is the point. A benchmark whose "long horizon" is 30 interactions and one whose "long horizon" is ten expert days are both in the literature under the same phrase. Where papers do fix a threshold, the thresholds disagree by orders of magnitude:

109 benchmarks of 995 — 11.0% — state a definition outright. The remaining 267 that define the construct do it by operational threshold (145) or by contrast with a shorter setting (122). A reader looking for a citable definition of "long horizon" in this literature will find one in roughly one paper in ten.

The same gap shows up from a second direction. A controlled ablation in the corpus reports that memory benchmarks under-test the very construct they name:

A controlled ablation on the LoCoMo benchmark shows that Bayesian belief updating alone provides little benefit over naive last-write-wins because existing conversational memory benchmarks rarely contain contradictory or differently reliable evidence.

The same paper measures a 27.5-point discrepancy between strict token-F1 and LLM-as-judge on identical outputs, which is a metric-validity problem in the same family: the construct is under-specified and the measurement of it is unstable.

304 of 995 artifacts (31%) are method-origin — a benchmark shipped alongside a memory method rather than built as a benchmark-first contribution — and 24 ship no artifact at all, reusing LoCoMo, ALFWorld or an existing dataset. Both are kept and flagged rather than dropped, so benchmark independence stays readable. 168 rows are flagged unverified: the released artifact could not be confirmed from the text.

Sorted by citation count. Search matches name, description, construct, setting and definition text. Every row links to the paper. The full table with evidence spans and warrants is in data/catalog.csv, which also carries the verbatim evidence span for every row.

The "What it is" column gives each benchmark twice: first the authors' own sentence, copied verbatim from the paper — 920 of 995 (92.5%) had one, 849 from the abstract and 78 only findable in the body — and then a plain-language summary written here. Where a source never characterises its own artifact, the cell says so rather than inventing a description; 51 rows fall back to a curated-list line and 12 have nothing but a title. Those weaker cells are marked in a different style.

Showing the first 300 matches here. The full catalogue — all 995 rows, newest first, with year/setting/construct filters — is a separate page: Agent Memory Benchmark Catalogue (also catalog.html in this package). Raw data: data/catalog.csv.

Estimated recall is about 80%, and 80% is the optimistic read. Two independent modality pairs agree: citation-snowball × human-curated lists estimates a population of 1,236 ± 161; keyword sweeps × human-curated estimates 1,279 ± 123. Against 995 captured, that leaves roughly 250–290 in-charter artifacts uncaptured. Capture-recapture estimates are lower bounds under heterogeneous catchability, so the true recall is likely below 80%, not above.

A third estimator was computed and discarded: S2-sweep × arXiv-sweep returned a population estimate of 728 against 993 observed. An estimate below the observed count is the signature of two samples that are not independent — both sweeps run the same keyword culture over overlapping indexes. It passed the standard overlap gate, which is precisely the case that gate cannot see.

• Every paper-finder modality is missing. The retrieval agent was auth-blocked for the entire acquisition phase, substituted with raw S2 and arXiv sweeps plus a hand-rolled citation snowball. This is the largest structural hole.
• The gap is localised, not diffuse. Filtering 33,275 citation edges to works that are both benchmark-shaped and memory-shaped but never captured leaves 48 cited by ≥4 core papers. Reading them: temporal-reasoning benchmarks are systematically under-captured (EvolveBench, TIQ, ComplexTempQA), as is lifelong/embodied navigation (GOAT-Bench, long-horizon VLN) — and one literal miss, Evaluating the Long-Term Memory of Large Language Models, cited by four core papers.
• Reference-pool recall is unusable here and is reported as gated. It reads 50.8%, but the top never-captured references are GPT-4, Llama 3, ReAct, Chain-of-Thought and MMLU. A benchmark corpus cites mostly models and methods; those absences are correct.
• The no-paper stratum is captured but not estimable. 37 artifacts have no academic paper at all; citation-graph signals cannot see them, so no estimate covers their recall.
• 13.9% of rows were read at abstract depth only (no arXiv id, so no reachable body). Those rows carry visibly thinner evidence and are reported as a separate stratum throughout.
• Judge calibration leans one tier high on the charter boundary. 98.4% same-side agreement over 125 salt judgments, but every exact-tier miss sits on two adjacent-family artifacts and 15 of 17 are one tier high. The "in" count is an upper read of that boundary.

Numbers in this section trace to coverage-verdict.md and scratch/mention-shadow-gaps.json in the run directory.