I have a little server on my Mac Mini called brain-api. I can save anything to it from my phone — a link, a restaurant somebody mentioned, a half-formed idea — and ask it questions later in plain English, and it answers with a citation back to the exact note.
Here's the honest scoreboard after a couple of weeks. My own store has 3 notes in it. Three! The other store on the same server, the one written entirely by my coding agents while I'm asleep, has 150.
That gap is the actual story here. The knowledge base that turned out to matter is the one I never had to remember to write.
The store is just markdown files
Everything lives in ~/memory/brain as plain .md files, managed through basic-memory. There's a SQLite index for search on top, and I treat that index as disposable — delete it and it rebuilds from the files.
That's the decision I'd defend hardest. The files are the truth. I can grep them, they back up with everything else, I can open the folder in Obsidian, and I could walk away from all of this tooling tomorrow without losing a thing. If your notes live in a database you can't read with cat, you don't own them, you're renting them.
Capturing
POST /capture takes text, a URL, a shared web page, whatever, and writes a markdown note into inbox/. On the phone it's an iOS Shortcut, so saving something is a share-sheet tap, an Action Button press, or "Hey Siri, save to brain." Friction has to be basically zero or I won't do it.
It responds before the note is actually finished, though. On purpose.
The enrichment pass runs after you've walked away
The moment the note lands, the server fires off a detached background pass and stops caring about it. That pass re-reads the note and asks Haiku to pull structured knowledge out of it against a schema:
{
"tags": ["3-6 lowercase topical tags"],
"observations": [
{
"category": "restaurant",
"content": "Nari — Thai, in SF, recommended by Jake"
}
],
"relations": [{ "relation_type": "recommended_by", "target": "Jake" }]
}That gets appended to the note as typed observation lines and [[wikilinks]]. So something I mumbled into my phone in twelve sloppy words comes out with a category, a person attached, and an edge to an entity note. The store cross-links itself and I never tidy anything.
Two things I like about that. It's off the write path, so capture stays instant no matter how slow the model is. And the failure mode is safe — if enrichment blows up, the raw note is untouched. Worst case I've got my note, unenriched, which is what I'd have had anyway.
Asking
POST /ask takes a question and returns a structured object:
{
"answer": "string",
"citations": [
{ "title": "…", "permalink": "…", "quote": "the supporting line" }
],
"confidence": "high | medium | low | not_found"
}Two deliberate bits there. Each citation carries a quote — the actual line that backs the claim — so checking an answer is a glance instead of a file hunt. And not_found is a first-class value in that enum, which matters more than it looks. Give a model an easy, legal way to say "your notes don't cover this" and it'll take it. Leave it out and it hands you something plausible instead, and now you've got a tool you can't trust.
Retrieval happens in my code, not in an agent loop. The server grabs the relevant notes and makes one model call with them pasted in. Under 25 notes it just sends all of them, because at that size the best retrieval strategy is no retrieval strategy. Above that it flips to hybrid search — vector similarity and keyword, top 8. That keeps the round trip under about ten seconds, which matters because iOS Shortcuts gives up on you at around sixty.
The second store, the one that fills itself
/ask takes a project parameter, whitelisted to two values so nobody can point it at some random folder on my machine. The second one is where this got fun.
Every night, a job called molt reads my Claude Code session transcripts and summarizes each one into this same kind of store: one note per session (133 of them), plus a rollup per project (17). The rollup is a running list of [left_off] lines linking back to the sessions they came from. Real entries from projects/revcogs.md:
[left_off] 2026-07-16— Auto-pricing is live in prod (both commits deployed, encryption/auth path proven against a real credential row) but zero listings have it enabled yet — feature is live but inert.
[left_off] 2026-07-16— All work implemented and passing tsc/lint/tests plus a DB roundtrip, but nothing was visually verified — a Chrome window was left open at the sign-in page, and nothing was committed to git.
I didn't write either of those and I don't remember either moment. Both are exactly what costs me twenty minutes of re-deriving when I wander back into a repo a week later.
So there's a third shortcut called Ask Code pointed at project=claude-sessions. I can stand in a grocery line and ask "what did I leave unfinished in revcogs?" and get a real answer with links back to the sessions. That still delights me a little.
It was about ten lines of change, too. Same retrieval, same prompt, same schema, different folder.
The gotcha that cost me the most
The Ask shortcut showed nothing for a while. No error, no empty string, just a blank screen. It was a Get Dictionary Value step trying to pull answer out of the JSON, failing silently on-device.
The fix wasn't to debug the Shortcut. It was a ?format=text param so the server hands back plain prose with nothing to parse. Dumbest client, dumbest response. There's a line in the README now telling future me not to make that shortcut "cleaner," because I will absolutely try.
Locking it down
This server holds my notes, my agents' notes, and a model with read access to both. So it's tailnet-only, over tailscale serve and never funnel. Serve is my devices, Funnel is the entire internet, and they're one word apart in a command you type once and then forget you typed. A bearer token sits behind that as a second layer, and the server refuses to start at all if it's missing.
Where it stands
My personal capture habit is still bad. Three notes. Not going to pretend otherwise. What's working is the half I don't have to maintain — my agents generate the knowledge, molt turns it into notes every night, and the same little API answers questions about both stores.
What I want next is the loop closed the other way. An agent starting work in a repo should read its own project rollup first, so the first thing it knows is where the last session left off. Right now that memory only flows toward me. It should flow back.