Memory: short-term and long-term

An agent without memory is a clean-slate predictor: every call starts from scratch, no continuity between turns. Most useful agents need at least:

1. Working memory — this conversation. Bounded by context window.
2. Long-term memory — facts learned in earlier sessions, retrieved when relevant.

These map onto two completely different mechanisms:

layer	mechanism	retrieval	persisted?
working	append to message list	sequential	no (in-process)
long-term	embed + vector search	similarity	yes (disk / DB)

References: - MemGPT (Packer et al., 2023) — formalises the two-tier idea with explicit memory paging - Generative Agents (Park et al., 2023) — memory stream + reflection, the canonical research demo

Tip⏱️ Session S6.3

One session (epilogue): read and run, then overflow the memory on purpose. Cards, prerequisites, and done-when tests: the Session Guide.

1. Working memory: a bounded message list

ConversationMemory is a deque of messages with a max size. When you exceed the limit, the oldest message falls off the end.

Why not just keep everything? Context window is finite and expensive. A 100k-token conversation costs ~$0.30 per turn on Sonnet just for the input. Past some point, old turns aren’t worth re-sending — but you can’t drop them naively either or you forget what the user told you 20 turns ago.

The trick: summarize before discarding. We support that via an optional summarizer LLM.

import sys
sys.path.insert(0, '../../src')

from ai_playground.agents.memory import ConversationMemory

mem = ConversationMemory(keep_turns=4)
mem.add('user', 'My name is Greg')
mem.add('assistant', 'Hi Greg')
mem.add('user', 'I work in ML')
mem.add('assistant', 'Cool')
mem.add('user', 'What is my name?')

# Only the last 4 turns are kept — the first ('My name is Greg') is gone.
for m in mem.messages():
    print(f'  {m["role"]}: {m["content"]}')

print('\nNotice: without summarization, the agent has now forgotten my name.')

With a summarizer

If we pass an LLM backend as summarizer, evicted messages get rolled into a running summary prepended to the message list. Memory of older facts survives at lower fidelity.

from ai_playground.agents.llm import LLMResponse

# A fake summarizer that just concatenates — for real use, plug in ClaudeBackend.
class FakeSummarizer:
    def complete(self, messages, **kw):
        # Pull the 'New turn to incorporate' part out of the prompt and append it.
        prompt = messages[0]['content']
        return LLMResponse(text=prompt.split('Rewrite')[0][-200:].strip())

mem = ConversationMemory(keep_turns=2, summarizer=FakeSummarizer())
mem.add('user', 'My name is Greg and I work on agents')
mem.add('assistant', 'Nice to meet you Greg')
mem.add('user', 'I prefer Python over JS')
mem.add('assistant', 'Noted')
mem.add('user', 'What do you know about me?')

for m in mem.messages():
    role = m['role']
    content = m['content'] if isinstance(m['content'], str) else str(m['content'])
    print(f'  {role}: {content[:80]}')

In production, you’d want token-aware truncation (count actual tokens with the model’s tokenizer, evict when over budget) and a smarter summarizer prompt. But the pattern is the same: bounded recency + lossy long-term compression.

2. Long-term memory: a vector store

For things that should outlive the conversation — user preferences, facts learned, past task outcomes — we need a separate store. The standard pattern is embed + similarity search:

1. Embed each memory into a fixed-size vector when it goes in.
2. Embed the query at retrieval time.
3. Return the top-K most similar by cosine distance.

Our VectorMemory uses a toy hashing-trick embedding by default (no extra deps, works offline). Swap in real embeddings (Cohere, OpenAI, sentence-transformers) for actual semantic recall.

from ai_playground.agents.memory import VectorMemory

store = VectorMemory()
store.add('User prefers Python over JavaScript', metadata={'topic': 'preferences'})
store.add('User is working on an LLM agent project', metadata={'topic': 'context'})
store.add('The capital of France is Paris', metadata={'topic': 'general'})
store.add('User has a deadline next Tuesday', metadata={'topic': 'schedule'})

print(f'Total memories: {len(store)}\n')

for query in ['what languages does the user know?', 'when is the deadline?', 'who is working on what?']:
    print(f'Query: {query!r}')
    for score, entry in store.search(query, top_k=2):
        print(f'  ({score:+.3f}) {entry.text}')
    print()

Notice the lexical embedding’s limitation: “languages” doesn’t lexically overlap with “Python”, so the ranking won’t reflect the semantic relationship. Real embeddings would handle this. A small upgrade to sentence-transformers/all-MiniLM-L6-v2 (90MB, runs on CPU) would make these queries work as you’d expect. The shape of the API stays identical — only the embed_fn changes.

3. Putting it together in an Agent

The Agent class takes both memories. Long-term store is queried before every user turn; relevant hits get prepended as a system note (a baseline RAG pattern). Working memory tracks the current conversation as usual.

from ai_playground.agents import Agent, ToolRegistry
from ai_playground.agents.llm import LLMResponse

class ScriptedLLM:
    def __init__(self, responses):
        self.responses = list(responses)
    def complete(self, messages, tools=None, system=None, max_tokens=1024):
        print(f'  [model sees {len(messages)} messages]')
        for m in messages[:3]:  # show first few
            c = m['content'] if isinstance(m['content'], str) else f'<{len(m["content"])} blocks>'
            print(f'    {m["role"]}: {str(c)[:100]}')
        return self.responses.pop(0)

# Pre-populate long-term memory with something the user 'told us' in a past session
long_term = VectorMemory()
long_term.add('User prefers concise answers, no bullet lists')
long_term.add('User is working on building an agent from scratch')

llm = ScriptedLLM([
    LLMResponse(text='You are building an agent from scratch.', stop_reason='end_turn'),
])

agent = Agent(llm=llm, long_term=long_term)
result = agent.run('What project am I working on?')
print(f'\nAGENT: {result.final_text}')

Notice the model saw a [relevant memories] message prepended to the user query — that’s how recalled context gets injected. The retrieval happens automatically every turn.

4. Persistence

VectorMemory(path=...) saves to JSON on every add(). Reload by constructing with the same path.

from pathlib import Path
import tempfile

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / 'memories.json'
    
    store1 = VectorMemory(path=path)
    store1.add('User likes RoPE positional embeddings')
    del store1
    
    store2 = VectorMemory(path=path)
    print(f'Reloaded {len(store2)} memories from disk')
    for score, entry in store2.search('positional embeddings', top_k=1):
        print(f'  {entry.text}')

5. When to use what

situation	use
single short conversation	ConversationMemory only
long conversation, can’t lose old context	ConversationMemory with summarizer
facts that should persist across sessions	VectorMemory
both	Agent(memory=..., long_term=...)

Anti-patterns: - Dumping every turn into vector store and retrieving everything every turn — defeats the point of bounded context. - Storing very short snippets (“yes”, “ok”) in vector memory — embeddings are noisy at that length, retrieval gets random. - Not refreshing summaries — old summary points stay forever, even when contradicted. Periodically re-summarize from scratch if the conversation has shifted.

Key takeaways

• Two memories, two mechanisms: sequential bounded list for the active conversation; embed-and-search store for cross-session recall.
• Summarize, don’t drop: when working memory overflows, compress evicted turns rather than losing them.
• Embedding quality is everything for vector memory: the hashing-trick default is for offline demos; use real embeddings in production.
• RAG-into-context is a baseline, not a ceiling: more sophisticated systems use tool-call-style retrieval (“the model asks the memory tool”) or page-in/page-out (MemGPT).

---

🔨 From-scratch project: p6 — An agent from a blank file

You’ve now traced the loop, the tools, and the memory. The from-scratch project is to write your own ReAct loop — parse, dispatch, observe, repeat — against a scripted LLM (provided; no API key), with the three things that separate a loop from a demo: a correct transcript, tool failures fed back as observations, and a step budget that actually halts.

Brief and rules: projects/p6_agent/README.md

uv run pytest projects/p6_agent/ -v