ModelBrain retrieval benchmarks: the locked test run

The metric is recall_any@5: a question counts as a hit when at least one gold source is among the five items returned. The reranker mode is the shipped one: a small cross-encoder reranks the top 20 candidates. The slots column is a bench-only variant that adds standing and date slots; the shipped daemon does not use it.

Medians with the 95th percentile in brackets, in milliseconds. The LoCoMo and BEAM 100K rows were timed three times and stayed inside our spread limits (slowest median within 10 percent of the fastest, slowest 95th percentile within 25 percent). Reranker rows used 4 threads, the count the shipped program picks on this Mac, except LongMemEval, which comes from the locked run at 8 threads. The top-five lists in every repeat were identical to the locked run on all rows checked.

What is timed: a recall call per question, not ingestion. For the plain mode that is the wide pool fetch the runner makes, an upper bound on the shipped five-result call. No cache was used. Full report.

On BEAM 500K the standing and date slots made nothing better, and the reranker mode scored slightly below the plain mode (0.587 against 0.599), so on that benchmark the reranker did not help. On LoCoMo the plain mode (0.665) sits well below the reranker mode (0.767), so the reranker is where most of the LoCoMo score comes from. On LongMemEval the three modes are within one question of each other.

BEAM's questions ask about things the user told the assistant: a plan, a preference, an earlier decision. The messages that hold those answers are the user's, not the assistant's replies. ModelBrain adds a fixed score of 0.012 to a candidate that was tagged as a user turn, after the other ranking steps have finished. On untagged data it does nothing. Our thesis is that this one prior, and not a stronger retriever or a cross-encoder, is the largest single reason for the BEAM scores. We did not run a no-bonus arm on the test splits, so the evidence is the development splits, where we did.

The bonus added 0.05 to 0.10 on BEAM dev and exactly zero on LoCoMo, whose records carry no role tags. LongMemEval records carry no role tags either. We read this as a reason to treat the BEAM scores as the result of a sound idea for chat-shaped data, and not as proof of a better retrieval engine.

One more limit: in the shipped product the bonus stays inert until a caller passes the role:user tag on a memory. These BEAM numbers describe a benchmark run where the formatter supplied the tag, not default behaviour for an untagged caller.

• Written first. The preregistration fixes the splits, the metric, the configurations, the claims rule and our own predictions before any test score existed. It carries dated amendments, including our own mistakes.
• Dev and test are separate. We tuned on development splits only: LoCoMo conversations 0 to 2, 13 BEAM 100K chats, 21 BEAM 500K chats, 100 LongMemEval questions. The test splits were LoCoMo conversations 3 to 9, 7 BEAM 100K chats, 14 BEAM 500K chats and 370 LongMemEval questions.
• One run per row. ModelBrain ran once from engine commit e945b16c3a1d78e167dd96e95bc0141fbc952d04. The lock file holds hashes of the units files, the reranker model, the scripts and the binary.
• One incident. A check in our run script wrongly voided the LoCoMo stage because it read a field that is written after a settings reset. Every reranked query had a reranked result and zero errors. We fixed the check and restored the original rows unchanged rather than re-running them. No score had been read. It is logged in the preregistration.
• Our predictions. We predicted ranges before the run. The LoCoMo plain mode landed below its range (0.665 against 0.68 to 0.74) and the BEAM 500K reranker mode landed below its range (0.587 against 0.60 to 0.69). The other predicted numbers fell inside their ranges.

Tried on dev and not adopted: bge-reranker-base (about 8 seconds a query at first, too slow to ship), a jina turbo reranker and early MiniLM conversions whose scoring head had been damaged (they failed our parity check), reranker pools from 10 to 50, a language-model judge as a second stage, two ordering switches, a preference-slot gate, and the variants named rl, v2, rr50, a spread rule and frame stripping. Each of the last five moved BEAM by at most one question. The adopted pieces are the MiniLM reranker at pool 20, the 0.012 bonus, and, in the bench-only row, the standing and date slots.

• Retrieval, not answers. recall_any@5 asks whether a gold source appears in the top five. It is not answer accuracy. BEAM's own metric is judged by a language model; recall_any@5 is a retrieval proxy.
• Speed is one machine. Timings come from one Apple M2 Pro with 12 cores (machine record), idle when each row started. A reranker needs cores, so a machine with fewer will be slower than shown.
• Small BEAM splits. BEAM 100K has 126 test questions, so one question moves a score by 0.8 points.
• Date slot. The date-aware slot fires only when the benchmark supplies the date each question was asked. The shipped daemon does not take a question date yet, so we make no claim for it in the product, and the reranker mode runs with the slots off.
• Edge paths. With a reranker loaded, the user-turn bonus is applied after the cross-encoder only on pages that are reranked. On filtered recall it is applied in the plain path. If the reranker fails on a call, that call returns the plain page without the bonus and is counted as an error. No test row had an error.

• Scores for every row on all four benchmarks
• Latency report
• Preregistration with every dated amendment
• Lock file with commit and file hashes
• Machine record

Our everyday recall test, which grades answers and not retrieval, is on the recall test results page.