Agent memory without extra LLM calls: what it costs

"Memory without extra LLM calls" is a useful phrase and a confusing one. A coding agent is an LLM — a large language model — and every session already costs tokens. The real question is whether the memory tool adds another model call on top: to summarize a session, extract facts, write embeddings, rerank results, or consolidate notes.

That distinction decides the bill. A memory tool that writes a note the agent already composed may add almost nothing. A tool that runs a background model over every session can add calls you never see in the chat window.

Checked against each tool's documentation on 2026-10-26.

Where a model can enter#

A hook is a script the harness runs at a session event, and it is how most capture tools see the session at all.

A tool can be "zero-LLM" at one stage and model-assisted at another. Ask which row you are paying for.

What each tool actually does#

Claude Code#

Claude Code's auto memory — the notes the model writes for itself about a project — is written by the session model. The memory docs describe Claude deciding what is worth remembering and saving it as plain markdown while it works. The docs do not describe a separate memory service, a separate API key, or a per-note charge. The work happens in the session you are already paying for.

The cost shows up as context — the text the model can see in one session — and turns:

• The first 200 lines of MEMORY.md, or 25KB, load at the start of every session. That is a fixed context cost whether or not you use the fact.
• Writing a memory is a tool call, so it consumes tokens in the turn where Claude writes it.
• Topic files are read on demand, which costs only when the agent opens one.

CLAUDE.md, Claude Code's instruction file, is cheaper to reason about: it is text you wrote, loaded at startup, with no model deciding what to keep. The trade-off is that you maintain it.

You can turn auto memory off for a project with autoMemoryEnabled: false, or with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1. Then the only memory is the instruction file and whatever store you connect.

Codex#

Codex memories are the clearest case of a memory feature that uses models by design. The memories page says memories are off by default and generated from eligible prior chats in the background. It exposes two model settings:

• memories.extract_model overrides the model used for per-chat memory extraction.
• memories.consolidation_model overrides the model used for global memory consolidation.

The docs say Codex waits until a chat has been idle long enough, and it can skip a background pass when you are near a limit.

So Codex memories are not a free local index. They are a model-assisted feature that runs in the background, with controls to keep it from competing with your active work. If you leave memories off, AGENTS.md, Codex's instruction file, remains the zero-extra-call path.

The defaults are deliberately conservative. A thread must be idle for six hours before it is eligible (memories.min_rollout_idle_hours), be no more than 30 days old (memories.max_rollout_age_days), and the remaining rate-limit window must be at least 25% (memories.min_rate_limit_remaining_percent) before a pass starts. Codex processes at most 16 eligible chats (rollouts) per startup pass. Those settings keep background memory from competing with the work you are doing, and they explain why a memory can take a day to appear.

The feature flag is the global off switch: [features] memories = false. The per-chat /memories switches in the TUI or desktop app are narrower: one stops this chat reading memories, the other stops it becoming an input for a future pass. The app's Delete Codex memories is the documented way to reset the store.

ai-memory#

ai-memory has a real zero-LLM path, if you read the default carefully. The README says the default path uses zero LLM calls: capture, search and handoffs work with no API key. The usage docs say a substantive session end writes a rule-based summary page unless you opt into provider consolidation.

In our test of version 3.2.5, with no LLM provider and no API key, the server stored six observations, produced a session page, and returned a search hit for the page. The session page was metadata-heavy, not a distilled lesson. That is the honest shape of the zero-LLM path: it records and indexes; it does not write a thoughtful summary.

There is one default download worth naming. The first start fetches a local embedding model, about 87 MB, from Hugging Face (local embeddings). An embedding model turns text into vectors so search can compare meaning; this one runs on your machine. The data handling docs describe the model file as a download, not a copy of your notes. Set embedding_provider = "none" to avoid it, and full-text search still works.

Consolidation is the model-assisted rewrite of stored notes into pages; lint checks them for problems. The opt-in paths are where the model bill appears:

• llm_provider configures the model used by consolidation, lint, bootstrap, dream, profile and auto-improve. Some are manual (bootstrap, lint); dream and SessionEnd consolidation are off by default. Consolidation sends session observations and page text; bootstrap sends repository sources (LLM providers).
• A cloud embedding provider sends the text of the pages it embeds. Switching to it can backfill the existing corpus.
• AI_MEMORY_RERANKER=llm sends each live query plus bounded titles and snippets.
• Admission webhooks send every write's body to a URL you configure (admission webhooks); a webhook is an HTTP callback to that URL.

The local path is not literally model-free: local embeddings are a model. It is LLM-free and API-free. That is the distinction that matters for cost and for data leaving the machine.

Where the costs show up#

Costs show up as provider tokens or subscription quota for model work, context tokens for a startup index, local CPU and RAM for embeddings, and human time for review. We did not measure provider spend or summary quality; our ai-memory test ran with no provider, and we did not test Codex memories.

How to keep the model calls down#

For ai-memory:

1. Do not set llm_provider or llm_fallbacks.
2. Set embedding_provider = "none" if you want no local model download. Full-text search still works.
3. Leave AI_MEMORY_RERANKER unset.
4. Do not configure admission webhooks unless you need them.
5. Leave capture_assistant off. It is a double opt-in for a reason.
6. Use the rule-based session summary, or have the agent write the page with memory_write_page.

For Claude Code:

1. Keep CLAUDE.md short. The target is under 200 lines; a long file competes for the same context you are trying to preserve.
2. Turn auto memory off for a project where you do not want the model deciding what to keep.
3. Do not add a memory MCP server unless you need one. MCP is the protocol an agent uses to call an outside tool, and each server adds a tool catalog to the context.

For Codex:

1. Leave memories off if AGENTS.md is enough. They are off by default.
2. If you turn them on, set memories.extract_model and memories.consolidation_model deliberately, and watch the rate-limit threshold.
3. Use memories.disable_on_external_context to keep tool-heavy chats out of generation.

When paying for synthesis is worth it#

Pay for model-assisted memory when the raw material is too large to curate: many sessions, long tool output, a team that will not maintain an index, or a need to merge overlapping notes. Stay zero-LLM when the decision is small and exact, the repository is sensitive, or you cannot audit a provider call. The middle path is to let the agent write the memory with its own model, as Claude Code does; you pay in session tokens, not a separate memory bill.

Where Noteheld fits#

Noteheld is the zero-LLM contrast: no server-side model, deterministic summaries, and the agent supplies the summary. The plan is per workspace, not per token, but the agent still spends tokens writing and reading. The trade-off is no synthesis: it stores and searches what the agent wrote and does not turn ten sessions into one lesson. If synthesis is the job, this is the wrong tool.