That's the only question worth asking about a memory tool. So we built a test that tries to catch our own mistakes, and we'll publish what it finds whether or not it flatters us.
In plain terms, the test checks four things: that the right note comes back, that the answer points you at the note it actually used, that both still hold when there are a thousand notes instead of ten, and that anything you deleted stays gone.
It's also the release gate. If the test fails, the first version doesn't come out.
The methodology is published now. No scores are published yet, and the runnable harness ships with the first release. We'd rather show you an empty results table than numbers from a prototype run we wouldn't trust ourselves.
Two versions of a decision exist and only the newer one is right. Returning the old one scores zero, even though it reads like a perfect match.
Same vocabulary, different answers. Tests whether your contexts stay separate or bleed into each other.
Questions whose best match was deleted. The correct behavior is to find nothing, and finding nothing is a pass.
Questions with no answer in your memories at all. Does it say so, or hand you its nearest guess with a citation attached?
Reference numbers, file names, addresses. Similarity search is bad at these, so something else has to carry it.
You ask in wording that appears nowhere in the note. Keyword matching is useless here; meaning has to carry it.
Did the objects that actually contain the answer appear in the returned set? Scored against a human-labelled gold set per query, not against embedding similarity.
Does the object cited contain the claim made from it? A right answer citing the wrong object is a failure, because the citation is the user's only way to check.
The same query set against vaults padded with 10×, 100× and 1000× irrelevant-but-similar memories. The reported figure is degradation, not the absolute score.
After a forget, does the content stay out of every later recall — including after a compaction pass and a daemon restart? An unrequested reappearance is a hard fail, not a scored metric.
Retrieval in isolation is half the question, so the harness also follows real multi-session tasks across three conditions: no memory, a plain maintained notes file, and ModelBrain. A notes file is a real baseline, not a straw man. If it wins, that's the finding.
$ mb-eval run --suite adversarial --vault ~/brain \
--noise 10x,100x,1000x --out results/
conditions none | notes-file | modelbrain
reports evidence@k, citation fidelity, degradation curve,
repeated explanations avoided, stale-memory mistakes,
token cost per task, user corrections needed
Illustrative commands. The harness is published alongside the first release, and it accepts any MCP-compatible memory backend, not only ours.
The table lands here when the first release does, with the run configuration, the gold set and the raw output beside it. Until then this section stays empty on purpose.
We wrote both the system and the test. A gold set authored by the people who tuned retrieval is a real bias, and publishing the set for others to contradict is a partial mitigation, not a fix.
Citation fidelity depends on the assistant's behavior as well as ours, so results are reported per host. Changing host model can move the numbers with nothing in ModelBrain changing.
Synthetic noise padding is not the same as a genuinely messy year of real work. It's what we can reproduce today, and it probably makes degradation look tidier than it is.
How full-text and vector ranking are combined is still an open question in the spec. Any score is a measurement of one recorded configuration, not a permanent property of the product.
If you run the harness and we come out worse than something else, send the configuration and the output. We'll publish the result and what we changed because of it.