Perfect Recall, Rising Cost

Keeping every word of the conversation, and understanding exactly what that costs you.

Prerequisites: You know what a language model does and have called one at least once.

The Problem

A language model has no memory between calls. This is the single most surprising fact for people new to building agents, because the products built on top of models hide it so well.

Here is what actually happens. You send a message. The model reads it, produces a reply, and returns it. Then the model forgets. Completely. Not gradually, not partially. The next call starts from nothing. If you send “My name is Alice” and then send “What is my name?”, the second call arrives at a model that has never heard of Alice.

Try it once without any memory layer and the effect is comic:

You:   My name is Alice and I work on agent systems.
Agent: Nice to meet you, Alice.

You:   What do I work on?
Agent: I don't have that information. Could you tell me?

The agent is not being difficult. From its side, the second question arrived alone, with no history attached. Every turn is a first turn. The illustration below shows the shape of the problem: two calls, no thread between them.

Two separate exchanges with a language model shown side by side, each arriving alone with an empty slot where the previous conversation should be

Something has to carry the conversation forward, and that something lives in your application, not in the model.

The Core Idea

Conversation buffer memory is the simplest possible answer: keep every message, and send all of them on every call.

Think of a court stenographer. They type every word spoken in the room, without judgment about what matters. When a lawyer asks “what did the witness say about the timeline,” the stenographer does not summarize or recall from memory. They read the transcript back from the beginning. The record is complete and the reading takes as long as the record is long.

A stenographer’s transcript unspooling from the machine, with a lawyer reading the full roll from the top while the room waits

That is buffer memory. There is no cleverness in it. The agent appears to remember because the entire past is placed in front of the model again, every single time. This gives perfect recall, and the cost of that recall grows with every word spoken.

The model does not remember the conversation. Your application remembers it and re-reads it aloud on every turn.

How It Works

Buffer memory has one data structure and one rule. The structure is an ordered list. The rule is: append, then send everything.

Step 1: Store Each Message With Its Role

Each entry in the list has two fields. Role says who spoke: user for the person, assistant for the agent. Many systems add a third role, system, for standing instructions that sit at the top and describe how the agent should behave. Content is the text itself.

The roles matter more than they look. The model uses them to reconstruct who said what. A list where the roles are wrong, or where two user messages appear in a row without a reply between them, produces confused output. Correct alternation is part of the data structure, not a formatting detail.

One turn is one user message plus the reply it produces. The list therefore grows by two entries per turn. After ten turns, twenty entries.

Step 2: Append, Then Send the Whole List

When a new user message arrives, it is appended to the end of the list. The complete list, from the first message to the one just added, is sent to the model. The model reads all of it and produces a reply. That reply is appended too, and the system waits for the next message.

The diagram below traces three turns. Notice that the arrow into the model gets thicker each time: turn one sends one message, turn two sends three, turn three sends five.

Three consecutive turns with the message list growing by two entries each turn, and the arrow feeding the model widening from one message to three to five

Algorithm Sketch: Buffer Memory Turn

1. Receive the user message
2. Append it to the list with role "user"
3. Send the entire list to the model
4. Receive the reply
5. Append the reply to the list with role "assistant"
6. Return the reply to the user

That is the whole technique. It can be built in an afternoon, and almost every agent starts here.

Step 3: Save the List If You Want It Tomorrow

The list lives in memory while the process runs. Restart the process and it is gone. Writing it to a file or a database on each turn, and loading it when the conversation resumes, gives the agent continuity across restarts. This is the humblest possible form of persistence, and it already puts you ahead of most demos.

Note what it does not give you. Reloading the list restores one specific conversation. It does not let the agent recall something the user said in a different conversation, and it does not scale past the point where the list stops fitting in a single call. Chapter 21 handles the first problem properly. The rest of this part handles the second.

Why It Breaks

Buffer memory has exactly one failure mode, and it is worth understanding precisely, because every technique in the next four chapters is a different response to it.

Cost Per Turn Grows Linearly

Everything in the list is re-read by the model on every call, and models bill for what they read. Turn one sends one message. Turn two sends three. Turn ten sends nineteen. The per-turn cost climbs in a straight line.

Total Cost Grows Quadratically

This is the part that surprises people. Because each turn is more expensive than the last, the running total does not climb in a straight line. It curves upward. A conversation twice as long does not cost twice as much. It costs roughly four times as much.

Put concrete numbers on it. Say each turn adds about 100 tokens of new text. At turn 10, you are sending roughly 1,000 tokens per call. At turn 50, roughly 5,000. But the cumulative spend across those 50 turns is around 125,000 tokens, not 5,000. The chart below shows both curves side by side: the straight line is what you pay per turn, and the curve behind it is what you have paid in total.

Two charts side by side: per-turn token cost rising as a straight line, and cumulative token cost rising as a steep upward curve

Then It Hits a Wall

The context window is the maximum amount of text the model can read in one call. Buffer memory grows without limit, so eventually the list will not fit. When that happens there is no graceful degradation. The call fails, or the system silently drops the beginning of the conversation, which is worse because it fails quietly and the agent starts contradicting things it agreed to an hour earlier.

There is no warning built into the technique. You get flawless recall right up until you get an error. The illustration below shows the shape of that ending: a container filling steadily with no valve, no overflow pipe, and a hard ceiling.

A vessel labeled with a fixed capacity filling with stacked message blocks that reach the rim, with no outlet and a solid ceiling above

Bigger context windows push the wall further out. They do not remove it, and they do nothing about the cost curve, which is usually the constraint that bites first.

When to Use This

Best for:

Short conversations, roughly under twenty turns, where the whole history fits comfortably
Prototypes and internal tools, where you want something working today and will optimize later
Cases where every detail genuinely matters and no summarization is acceptable, such as a transcript-driven workflow
Being the baseline you measure other techniques against

Overkill when:

Conversations routinely run long, which is most consumer-facing products
You are cost sensitive at any real volume
The agent needs to recall anything from a previous session

Tradeoffs:

Factor	Impact
Latency	Low at first, then rising as the model reads more each turn
Complexity	Lowest of any technique in this book
Cost	Linear per turn, quadratic across the conversation
Recall quality	Perfect, until the context window overflows and it becomes zero

Compared to doing nothing: An agent with no memory layer cannot hold a conversation at all, so buffer memory is not optional, it is the floor. The real question is never whether to keep history but how much of it to keep and in what form. Every remaining chapter in this part answers that question differently: keep only the recent turns, replace old turns with a summary, do both, or trim to an exact budget.

Key Takeaways

Language models are stateless between calls. Memory is something your application provides, not something the model has.
Buffer memory keeps every message in an ordered list of role and content pairs, and sends the whole list on every call.
Roles are load-bearing. The model reconstructs who said what from them, and broken alternation produces confused replies.
Cost per turn grows linearly with conversation length, and total cost across the conversation grows quadratically.
The failure mode is abrupt: perfect recall until the context window fills, then an error or silent truncation with no warning.
Every technique in the next four chapters is a different answer to the same question this one raises: what do you drop, and how do you choose?

Companion Notebook

The working implementation of this technique is in conversation_buffer_memory.ipynb in the Agent Memory Techniques repository.