An agent without memory meets every user as a stranger and relearns your project every session. An agent with careless memory is worse: it remembers the wrong things, confidently, forever. Persistent memory has become one of the most active areas in agent tooling this year because teams are moving agents from one-off scripts to long-term collaborators. This article is about designing it so it helps.
Three kinds of memory, and why the distinction matters
It helps to borrow terms from psychology, because the three kinds behave differently and need different handling.
- Semantic memory — facts. "The user deploys to AWS Mumbai." "This repository uses pnpm." Small, stable, and valuable to retrieve often.
- Episodic memory — what happened. "On Tuesday we tried caching the embeddings and it made results stale." Useful for not repeating mistakes; loses value with age.
- Procedural memory — how to do things. "When releasing, run the smoke tests before tagging." This is often better written as a skill, under review, than learned silently by the agent.
The write policy is the whole design
Storing text in a vector database is the easy part. The decisions that determine whether memory helps are about what gets written in the first place.
A useful rule: write a memory only if it would change a future decision. A user mentioning they are tired today does not; a user saying they cannot use paid APIs does. Most agent turns should write nothing.
- Extract, do not transcribe. Store "prefers Python over TypeScript for scripts", not the paragraph it came from.
- Record the source and the date with every memory, so stale or disputed facts can be traced.
- Separate what the user said from what the agent inferred, and trust them differently.
- Write in the background where you can, after a session ends, rather than slowing every turn.
Facts change: handling updates and contradictions
The failure that hurts most is the agent acting on something that used to be true. The user changed jobs, the project moved from Postgres to MySQL, the team abandoned an approach. A memory store that only appends will hold both versions and retrieve whichever is more similar to the query.
Treat each new semantic memory as a possible update. Before writing, search for existing memories about the same subject, and if the new fact supersedes one, replace or mark the old one rather than adding a second. Episodic memories can simply age: weight recent events higher and let old ones fall out of retrieval.
Retrieval: getting the right memory into the window
Memory only helps if the relevant item is retrieved at the right moment and the irrelevant ones are not. Retrieving too much recreates the context-window problem you added memory to solve.
- Retrieve a small number of memories per turn and include their dates.
- Combine similarity search with simple filters — this user, this project — so one user's memories never surface for another.
- Load a short profile of stable facts at the start of a session instead of searching for them every turn.
- Let the agent search memory as a tool when it needs to, rather than always injecting results.
Privacy, consent and deletion
Memory is personal data with a long life. If your users are in India, the Digital Personal Data Protection Act applies to it, and most jurisdictions now expect a person to be able to see what is held about them and have it removed.
Design for that from the first day: store memories per user with a clear owner, never mix them into shared embeddings, let users view and delete what the agent remembers, and do not store secrets, credentials or sensitive categories of data at all. A memory that leaks across users is not a quality bug; it is a breach.
Testing memory like any other feature
Memory is invisible until it fails, so write tests that span sessions rather than relying on demos.
- Recall: tell the agent a fact in session one, ask a question that depends on it in session two, check the answer uses it.
- Update: state a fact, contradict it later, confirm the agent uses the newer one.
- Isolation: confirm a fact from user A never appears for user B.
- Restraint: run a session of small talk and confirm nothing was written.
- Deletion: delete a memory and confirm it is never retrieved again.
A worked example: a study-planning agent
Consider an agent that helps a student prepare for AI engineering interviews over several weeks. In the first session the student says they have a final-year exam in December, can study about five hours a week, already know Python well, and find system design questions hardest. They also mention they are tired today and want a short session.
A good write policy stores four semantic memories — the exam month, the weekly hours, the Python strength and the system-design weakness — each with the date and the note that the student said it directly. It stores nothing about being tired, because that will not change any future decision. At the end of the session it writes one episodic memory: the student attempted two retrieval-augmented generation questions and struggled to explain chunking.
Three weeks later the student says they have an internship now and only two hours a week. The agent finds the existing memory about weekly hours and replaces it rather than adding a second, so the plan it builds uses two hours, not five. The episodic note about chunking has aged, so it ranks below the fresher note that the student has since practised it. When the student asks the agent to forget everything about their exams, the memories tagged with that subject are deleted and a test confirms none of them is retrieved again.
Where to keep memory
The store matters less than the policy, but the choice still shapes what is easy.
- A relational table is enough for a small profile of stable facts per user, and makes viewing, editing and deleting trivial.
- A vector index helps when there are many episodic memories and you need to find the relevant few by meaning; keep the user identifier as a filter, not just a field.
- Many teams use both: structured facts in a table loaded at the start of a session, and episodic notes in a vector index searched on demand.
- Whatever you choose, keep the write, update and delete logic in your own code, so the policy is testable rather than buried in a library default.
Key takeaways
- 1.Separate facts, events and procedures; they age and update differently.
- 2.Write a memory only if it would change a future decision — most turns should write nothing.
- 3.Handle contradictions on write, or the agent will act on facts that used to be true.
- 4.Memory is personal data: isolate it per user and make it viewable and deletable.
- 5.Test recall, update, isolation, restraint and deletion across sessions.
Do it now, free
AI Project Idea Generator
Five ideas, each with an architecture, a realistic number of hours, and how you would evaluate it — because the difference between a portfolio project and a tutorial clone is usually evaluation.
No sign-up. The internship is free to join too, with real project briefs and reviewed submissions.