Designing the recall test before writing the product

It is tempting to build the feature first and work out how to evaluate it later. Here is our own proposal: write the test before the feature, as a fixed public bar, so it cannot be written around what the product already does. Ours existed as a spec before most of what it measures did, and it scores no memory, a plain notes file and ModelBrain against each other on real multi-session tasks.

Why evaluate before you finish building

A memory system is easy to make look good in a demo and hard to make actually trustworthy under adversarial conditions: corrections, near-duplicate confusion, silence, deletion that doesn't really delete. If the test is written after the feature, it tends to get written around whatever the feature already does. We wanted the reverse: a fixed, public bar the product has to clear, decided before we knew whether it would clear it.

The five questions, as a real evaluation framework

The public version of this lives on our own five-questions checklist: does it handle contradiction correctly, can it tell similar things apart, does deletion actually delete, what does it do with silence, and where's the evidence. Internally these became a real, scored comparison, no-memory versus a plain notes file versus ModelBrain, on real multi-session tasks, measuring repeated explanations avoided, correct use of prior decisions, stale-memory mistakes, token cost, and how often a user has to correct it.

What this bought us

A number we can reproduce is worth more than a demo that only works once. It also means when we tell you a capability is “designed, not yet built” or “not yet tested at scale,” that's coming from the same harness we'd use to make a claim we're confident in, not a softer standard for the parts that aren't ready.