Designing the recall test before writing the product

It's tempting to build the feature first and figure out how to evaluate it later. We did it backwards on purpose: the recall test existed as a spec before most of what it measures did.

Why evaluate before you finish building

A memory system is easy to make look good in a demo and hard to make actually trustworthy under adversarial conditions: corrections, near-duplicate confusion, silence, deletion that doesn't really delete. If the test is written after the feature, it tends to get written around whatever the feature already does. We wanted the reverse: a fixed, public bar the product has to clear, decided before we knew whether it would clear it.

The five questions, as a real evaluation framework

The public version of this lives on our own five-questions checklist: does it handle contradiction correctly, can it tell similar things apart, does deletion actually delete, what does it do with silence, and where's the evidence. Internally these became a real, scored comparison, no-memory versus a plain notes file versus ModelBrain, on real multi-session tasks, measuring repeated explanations avoided, correct use of prior decisions, stale-memory mistakes, token cost, and how often a user has to correct it.

What this bought us

A number we can reproduce is worth more than a demo that only works once. It also means when we tell you a capability is “designed, not yet built” or “not yet tested at scale,” that's coming from the same harness we'd use to make a claim we're confident in, not a softer standard for the parts that aren't ready.