Memory & State

"Three layers:

• (1)Short-term — the message history array. This IS the memory for the current conversation. It's just text appended to the prompt each turn.
• (2)Long-term — persist key facts to a vector store after each conversation. Before the next conversation, retrieve relevant past context. Example: 'User prefers formal tone' or 'User's team uses Python 3.11.'
• (3)Structured memory — for critical facts, store in a database (not just vector). User preferences, past decisions, account metadata. The challenge: what to remember and what to forget. I'd use an LLM-based summarizer at conversation end to extract durable facts, and TTL-based expiry for ephemeral ones."

"Context management strategies:

• (1)Observation summarization — after each tool call, summarize the result to 2-3 sentences instead of keeping the full response.
• (2)Sliding window — keep the system prompt + first user message + last N messages. On current Claude models history is append-only, so the middle tool results are cleared server-side with context editing, not dropped by my code.
• (3)Context compaction — Claude's Agent SDK does this automatically: when approaching the limit, it summarizes older turns while preserving key facts.
• (4)Hierarchical delegation — offload sub-tasks to sub-agents with their own context windows. The orchestrator only sees summaries.
• (5)External scratchpad — write intermediate results to a file/DB, reference by ID instead of keeping in context."

“They’re three different things, and people use the words interchangeably, which is exactly why memory bugs are so hard to talk about.

The context window is a hard limit on the model — how many tokens it can attend to in one call. It belongs to the model, it costs money every call, and it is gone the moment the call returns. The model is stateless; nothing persists inside it.

Conversation history is what you choose to resend. It’s your buffer of prior turns, and it’s the thing that grows until it no longer fits. Windowing, summarising and eviction all live here.

Memory is what survives the conversation — preferences, facts, past decisions — stored outside the model and retrieved back into the window when it’s relevant. It is a database you own, with all the ordinary database problems.

The practical consequence is that ‘the agent forgot’ is three different bugs: it never wrote the fact, it wrote it but didn’t retrieve it, or it retrieved it and your history policy then trimmed it back out.”

“First I’d ask how it got there, because most bad memories are an inference written as a fact. The user asked about vegetarian restaurants once, and the agent stored ‘user is vegetarian’. That is a guess promoted to a permanent record, and no correction path fixes a write policy that does that.

For correction itself: memories have to be updatable and supersedable, not merely appendable. When the user says ‘I’m not vegetarian’, that has to supersede the old row — otherwise retrieval returns both and the model picks whichever ranked higher.

So I want three fields on every memory: where it came from, when, and whether it was stated or inferred. Then correction is a real operation — find the conflicting rows, mark them superseded, write the new one.

And the user needs to be able to see and delete what the agent thinks it knows. That’s a trust feature, and in a lot of jurisdictions it’s also a legal requirement.”

“There are two different problems hiding in that question.

If it’s two sessions of the same user — a phone and a laptop — it’s ordinary database concurrency and I’d treat it as such. Version every memory row, compare-and-set on update, and on conflict re-read and re-apply rather than overwrite. Last-write-wins is fine for a preference and catastrophic for a running total.

If it’s two agents writing shared state in a multi-agent system, the real answer is usually don’t. Shared mutable state between agents is what makes those systems impossible to debug. Give each agent its own scratch state, and have exactly one writer for anything shared — a coordinator, or an append-only log the others read.

And there’s a wrinkle specific to agents: a long run reads state at step one and acts at step seven. Whatever it read may be stale by then. So for anything that matters, re-validate at the point of action, not at the point of planning.”

“The request is ‘delete everything you know about me’, and the honest answer is that it lands in five places, not one.

The memory rows are easy. The vector index is not — deleting a row doesn’t remove its embedding unless the index really supports deletes, and plenty only tombstone, which means the vector still shapes the graph until you rebuild. Then the conversation logs, which contain the same facts in prose. Then traces and observability, which nobody remembers and which retain whole prompts. And then the model provider’s retention window, which is a contract question rather than an engineering one.

Which means the real design decision is upstream: tag everything with a subject id at write time, so erasure is a query instead of an excavation. If you can’t enumerate where a user’s data went, you can’t honour the request — and you’ll discover that while a clock is running.

And be careful about derived artifacts. A summary generated from deleted conversations is still derived from them.”

“First, why I can’t edit: on the newest models each thinking block is bound to the exact conversation before it. Delete or rewrite an earlier turn and every later block is invalid — a 400 on newer accounts, and a cache miss from the edit onward on any account. So the history my client sends only ever grows, and the shrinking happens on the server.

I use three features for three jobs. Context editing clears finished tool results and old thinking server-side, keeping the last few. Compaction summarises the conversation into one block when it crosses a trigger — 150K input tokens by default — and I append the whole response.content so that block survives. Memory writes anything the next session needs to storage I own.

The rest is what I choose to append. A huge tool result gets digested before it goes in, or read by a subagent in its own window. I never change the system prompt mid-run; I append a system message instead. And I freeze the tools at session start so the prefix stays cached.

The window being a million tokens doesn’t change this: every turn re-reads everything, so the cost and the attention problem arrive long before the limit does.”

“I split the replay into two kinds. Replaying a model call costs tokens. Replaying a tool can refund a customer twice. And every step has a window between the side effect and the record of it that I can’t close outside one database transaction — so the design is about making a replay inside that window a no-op.

Three rules. Save each assistant turn, verbatim, before running any tool in it. Key every side effect by the run id plus the tool_use id — stable on every replay because that turn was saved first — and pass it to the API, or check before writing if the API has no keys. And keep the transcript append-only in a run ledger with a lease, so exactly one worker can pick the run up.

Resuming is then the ordinary loop, started from the stored transcript. The last message tells me where I died: a tool call with no result gets re-run with the same key, a finished set of results means call the model, a final answer means stop. A started call to a tool that can’t take a key goes to a person.

And I persist the step and dollar counters with the transcript, or every crash hands the run a fresh budget.”