Here is our own proposal: a release that says it did not change recall should prove it question by question, by publishing how many questions return the identical five results in the identical order, rather than rerunning the benchmark and quoting a score that matches. ModelBrain's four 1.0 builds, the released one among them and one of which deleted nineteen failed ranking experiments and their code, each returned the same five results in the same order on all 1,942 of our benchmark test questions.
Our retrieval figures are recall_any@5, the share of questions where at least one correct record is in the top five, and recall_all@5, the share where every correct record is. Both are averages over hundreds of questions. Two kinds of change leave them where they were. A correct record that moves from first place to fourth still counts. And a gain on one question cancels a loss on another: fix question 12, break question 340, and the score does not move while two users get different answers than before.
So a rerun that prints the old score shows the average held. It does not show that the product a user installs answers the way the published figures describe.
Each benchmark run writes, for every test question, the five memories recall returned and their order. We keep the rows from the build whose figures are on the benchmarks page frozen, and every new build's rows are compared with them one question at a time. Each question lands in one of three buckets: identical order, the same five in another order, or a different five. Only the first bucket counts as unchanged.
The pass rule and the expected result are written into our preregistration file before the run starts. If every question is identical, the published figures carry over to the new build. If any question differs, the scores are compared against Mem0, Honcho and total-agent-memory, the release must still tie or beat each of them on every test set, and a loss means the change behind it is reverted rather than shipped.
The test questions come from four sets: LoCoMo (1,152 questions), BEAM at 100K tokens (140) and at 500K tokens (280), and LongMemEval (370). The scored recall figures on the benchmarks page count only the BEAM questions with an answer on file, 126 and 252; this check covers every question. Going from 0.6.8 to 1.0, we checked four builds this way.
Each returned the identical five, in identical order, on 1,152 of 1,152, 140 of 140, 280 of 280 and 370 of 370 questions. That is why the benchmarks page can say its figures describe 1.0 without a new benchmark run behind every claim.
It covers the recall path the benchmark exercises, which calls the store directly. It does not cover how an assistant phrases a question or reads the answer, and it does not rerun the LongMemEval answer reads: those carry over because the reader sees the same five results, not because they were read again. It also says nothing about questions outside these four sets. A build that changes ranking on purpose cannot pass this check, and should not try to: it needs a full benchmark run and its own figures.
The figures and their caveats: the benchmarks page. Why we publish a strict and a lenient score side by side: what "accurate" should mean for an AI's memory.