The Memory That Never Recalled

$ npm test
command not found: npq-hero

That's August 26, 2026, in my own repo, on my own machine.

Three weeks earlier I had written this down, in a file, on purpose:

zsh aliases npm to npq-hero (a package-audit wrapper), and the alias IS expanded in Claude Code's Bash tool, but npq-hero isn't on PATH there, so plain npm run ... fails with command not found: npq-hero. Workarounds: command npm ... (durable) or the full path.

The note is dated August 2, 2026. It is precise. It names the cause, the symptom, and two fixes. It even ends with an instruction to myself: state the workaround in every subagent prompt that runs shell commands here.

I had spent the previous four weeks running an experiment whose entire purpose was to make notes like that one reach me. It was still running that morning. And it sat there while I rediscovered a fact I already owned.

What keel is, and why memory was never keel's job

I build a thing called keel. It's a thin layer under Claude Code: guards at the ingest boundary, commit hygiene, an activity log, and wiring for a notes backend that stays mine. Plain files, synced however I choose. It's a personal tool built in the open, currently v0.6.

It stays small because it obeys two rules.

The first is own your config, buy your capabilities. Sync, guardrails, and setup are configuration: personal, unshippable, correctly mine, a few hundred lines. Memory, recall, and reflection are capabilities: general, valuable to strangers, and already built better by people who do it full time. keel wires those in. It does not reimplement them.

I learned that rule the hard way. Six separate times during design, the answer to "should I build this?" turned out to be "someone already did."

The second rule is documented surfaces only. If Anthropic hasn't documented it, keel doesn't read, write, or parse it. The system keel replaced parsed session transcripts out of an undocumented JSONL format. It worked, and it was the least durable thing in the stack; one release note mentions reducing transcript size up to 79 times over, which is exactly the kind of change that breaks a parser with no test to catch it.

So memory was never something keel was going to build. Memory is a capability. The question was only ever whether wiring one in earned its keep.

That's what I set out to measure.

The trial was deliberately done by hand

On July 28, 2026, I connected an agentic memory service and then refused to automate any part of it. No hook, no cron, no background worker. Every time a memory file changed, I told the agent to send it. Every time I wanted a fact back, I had to ask for it by name.

Here's the reasoning I wrote down that day:

whether a shared memory across machines is worth having is unproven, and packaging unproven things is how personal projects stall. Two weeks of using it by hand answers that.

The setup is two machines, two operating systems, one head. A fact I establish on the Windows box at 11 p.m. should be available on the laptop the next morning. That's the whole pitch, and I wasn't going to take it on faith.

My architecture notes already split a memory system into two operations. Recall happens before the turn and pulls relevant facts into context. Retain happens after the turn and stores what's worth keeping. One rule governs both: recall is local file reads, retain is remote work, never invert it. Recall sits in the path of every prompt I type, so it has to be fast, offline-safe, and incapable of failing in a way I notice. Retain can be slow, batched, and running on a machine that's currently switched off.

Both halves were manual. Both were supposed to be measured.

What held up

Retain worked, and the habit stuck. Fifty-five records over twenty-nine days, two of which turned out to be duplicates, though I'm getting ahead of myself. The practice was still live on the day it went under review, which is more than I can say for most things I try for a month.

The cost question was never a real question. At roughly a cent and a half per file, twenty-nine days of this came to under $1. I'd flagged cost as a thing to watch. It was noise, and watching it was wasted attention.

That's the good news, and it's genuinely good news. Then I opened the records.

A convention nobody enforces is a convention that drifts

The whole scheme rested on one rule: use the file path as a stable document id, so re-sending a changed file updates the entry instead of duplicating it. One rule. I put it in the instructions the agent reads at the top of every session.

Four weeks of a human executing that rule produced four different id schemes for the same class of file. Absolute paths. Paths relative to the config directory. Paths with the project segment silently truncated. And an entirely different convention, memory:project:slug, that I have no memory of inventing.

My journal entries alone arrived under three different prefixes. Some ids kept the .md extension; some dropped it.

None of that is cosmetic, because the id is the deduplication key. Two records were the same file written twice, under different ids, with different extracted content, both live, neither aware of the other. One of them was a machine inventory: a document whose entire job is to be the single authoritative answer to what hardware I own. There were two of it, and they disagreed.

The rule was never broken on purpose. Each session, an agent with no memory of the previous session's choice guessed at a path convention, and guessed differently.

Here's the part that stings. keel's own README already says this about its documented-surfaces allowlist: the allowlist is currently a written convention, not a runtime check. No code enforces it. keel doctor doesn't verify it. I wrote that sentence as a disclosure, an honest note about a known soft spot.

I did not read it as a warning about the next thing I built.

Recall never ran, and I'd already written down why

Go back to that npq-hero failure for a second, because it isn't really a story about npm.

The note lived in projects/-home-jimmy-mcp-ynab-mcp/memory/. I wrote it while working on an unrelated MCP server. The morning it would have saved me, I was in the keel repo. Claude Code natively loads the memory directory for the project you're in, which is a genuinely good default and which did exactly nothing here, because the fact was filed under a different project.

Cross-project recall is the entire reason I wanted a shared store. That morning the store held the answer, was online, was responding in about two-tenths of a second over Tailscale, and was never asked.

Nothing asked it. That's the whole failure, and it's not subtle once you say it out loud.

In the same paragraph where I set up the trial, I had written this:

manual recall is the weaker half of the test - asking "check hindsight for X" is cue-dependent by construction, which is the thing an unprompted recall hook exists to fix.

I knew. On day zero. And then I ran a four-week experiment whose primary variable I'd already identified as unmeasurable by the method I'd chosen.

It happened again, larger, near the end. I spent most of an afternoon in a long planning session about how to spend my time, and the store held two files describing precisely that: how my day is actually shaped, and what happens to me when too many external obligations stack up in one afternoon. Relevant is an understatement. They were never retrieved. The only time the memory service got queried in that whole stretch was when I explicitly told it to go look, so I could write this post.

My architecture notes cite a public test of a competing memory product where the agent had solved a problem the week before and only surfaced it after the user got frustrated and asked don't you remember? I quoted that as a cautionary tale.

Then I built a system that reproduces it exactly, and didn't notice for a month, because I was inside it.

A store the model may query is cue-dependent. A file already in context is not. I had that sentence in writing before I started.

The store held a fifth of what I know

I went looking for gaps and found that the gap was the store.

My local memory directories hold 262 fact files across 17 projects. The oldest is dated May 4, 2026, about twelve weeks before the trial began. After four weeks of diligent manual retaining, the shared store held 53 distinct documents. Set aside the journal entries and a couple of one-off notes, and roughly 45 of those correspond to an actual memory file.

Fewer than one in five.

The reason is mechanical and obvious afterward: I retained files when I touched them. The trial could only ever see the working set, whatever I happened to edit inside its own window. The working set of any given month is a thin and unrepresentative slice of what you actually know.

The specific losses are worse than the ratio suggests, because durable facts are by definition the ones that stop changing. Two of the files that never made it are from early May. One records how I lose a day when too many external obligations stack up. The other records the actual shape of my working day: when I eat, when I train, when I stop.

Those two change how an agent should behave in nearly every session I have. They haven't been edited since May, precisely because they were right the first time. So they were never touched. So they were never sent.

The facts most worth syncing are the ones a sync will never notice. A migration and a sync are different operations. I built the sync and assumed it would eventually add up to a migration. It can't. A change-triggered process converges on the churn, not on the corpus.

What I got wrong

I set out to answer whether a shared memory across machines is worth having. I didn't answer it.

Three of my four findings are about mechanism, not about the premise. Retain is cheap and sustainable. Recall doesn't happen without a hook. Unenforced conventions drift into duplicates. Syncing changes never populates a store. Only the first tells me anything about whether the idea is good; the rest tell me my harness was broken in ways that had nothing to do with the question.

I was measuring my own discipline and calling it a product evaluation.

The deeper mistake was starting without writing down what result would count as success. With no criterion fixed in advance, I evaluated on vibes. Vibes reported "this seems to be working" for four weeks straight while one of the two halves never ran at all.

And I violated my own first rule without noticing. keel buys capabilities and owns configuration. Memory was the bought capability, and it behaved fine. What failed was the configuration half, the part keel actually owns, held together by a written convention with nothing enforcing it. I outsourced the hard problem correctly and then fumbled the easy one.

The fix was not one commit, and that part was instructive too

The recall hook goes in. UserPromptSubmit is a documented surface with a documented field for injecting context alongside a prompt, which clears keel's second rule without an argument. It reads local markdown only, so there's no network in the path of a keystroke, and if it fails it returns nothing and the turn proceeds.

I built it, wrote nine tests, watched them pass, and then pointed it at the real corpus.

The first version was bad. Asked why did npm test fail with command not found, it confidently surfaced a note about end-to-end test topology. Asked whether I should book a contractor call, it said nothing at all, while a file about exactly that sat two directories away. Five realistic prompts, one useful answer.

The cause was boring and worth knowing: plain term overlap. Words like test and npm appear in half my corpus and carry almost no information, while npq appears in exactly one file and is practically a fingerprint. Weighting rare terms above common ones fixed most of it. Stemming fixed more, because fail and fails are the same question asked twice. Now that first prompt surfaces the npq-hero note at the top of the list, in about ninety milliseconds across all 262 files.

It still isn't right. Ask it when I stop working each day and it hands you something about my daughter, because daily and day don't stem to the same root and nothing in my scoring knows they're related. That class of miss needs embeddings, and embeddings need a service, and a service in the path of every keystroke is the exact thing my own rule forbids. So it stays imperfect on purpose, and I'd rather write that down than discover it again in November.

One more thing happened that I want on the record. Midway through the fix I broke the hook badly enough that it returned nothing for every prompt. It took me three rounds of probing to notice, because the hook is built to fail silently, and a hook that's working correctly is also silent most of the time. The failure mode and the success mode look identical from the outside.

That's the cost of designing something to degrade quietly. It degrades quietly at you, too. My tests caught it in under a second once I finally ran them, which is a lesson I apparently need to learn on a schedule.

What's left

The id becomes derived, never typed. A human retyping a path convention out of prose is the bug. The path is already known to the process doing the retaining; it should compute the id and never ask.

Then a one-time backfill, kept separate from the sync, covering every file rather than every file I happened to edit. And a deduplication pass first, because otherwise I'm building retrieval on top of two machine inventories that disagree.

This time I wrote the numbers down before starting. Recall fires on ninety percent of prompts. Five clear instances of a fact arriving unasked. Ninety-five percent corpus coverage. Zero new duplicates. I check on September 23, and the honest outcome is the one I publish.

The part that transfers

"Let me try it by hand before I automate it" is good instinct and incomplete practice.

Doing something by hand tells you whether you'll keep doing it. It does not tell you whether it works. Those two feel identical from the inside for about a month, which is roughly a month longer than you'd like.

The missing step costs ten minutes. Before you start, write down the number that would change your mind.