reference

Why LLMs forget

Language models forget between conversations because they are stateless: each request is processed on its own, and nothing carries from one session to the next unless something outside the model puts it back.

The question sounds like it is about a flaw, and it is really about a design. A large language model does not keep a running memory of its conversations. Every time it is called, it starts from the same fixed weights and reads only what is placed in front of it that turn. The sense that it has forgotten a decision made yesterday is accurate as an experience and misleading as a diagnosis: there was no store for the decision to fade out of. The gap comes from how the model is built and served, and it closes only when something outside the model keeps the record.

§01 · short answer

Each call starts from nothing carried over.

A trained model is a fixed set of weights. Those weights do not change as it is used; asking it a question does not write anything back into the model. So between one conversation and the next there is no place inside the model where a memory could be stored. Whatever the model appears to know about the current exchange, it knows because the text of that exchange is present in the request, not because it retained anything from before.

This is what statelessness means in practice: the model holds no state across calls. Two conversations run a day apart are, to the model, unrelated. The second one does not begin where the first left off unless the first one’s content is supplied again.

§02 · context window

What looks like memory — and where it stops.

Within a single conversation, the model does seem to remember: it can refer back to something said earlier in the same chat. That is the context window at work, the span of text supplied with each request. As the conversation grows, earlier turns are kept in that window and sent along, which is why continuity holds for a while.

The window has two limits that matter here. It is finite, so once a conversation outgrows it, the oldest content is dropped to make room, and detail in the middle of a long window is attended to less reliably than detail at its ends — a measured, repeated finding, not a quirk of one model. And it is per-session: the window belongs to one conversation and is not carried into the next. Context windows have grown large, into the hundreds of thousands and beyond, which raises the ceiling within a session but does not change the boundary at its edge.

figureWhere the answer sits decides whether it's found
illustrative
The same fact is easiest to retrieve at the very start or end of a long context and hardest in its middle — a measured, repeated finding, not a quirk of one model. A larger window widens the span but does not flatten the dip, which is why size alone doesn’t close the gap.Illustrative of the “lost in the middle” pattern reported by Liu et al., Transactions of the Association for Computational Linguistics (2024).
§03 · why it reads as forgetting

A contractor, not a colleague.

Picture the gap: a colleague who worked on a project last week remembers it this week, and picks up where things stood. A contractor hired by the day reads the brief each morning, does the work, and retains none of it tomorrow; the next day’s contractor is briefed from scratch. A stateless model is closer to the second. It can be excellent within the day and carries nothing into the next unless the brief carries it.

The frustration people describe, re-explaining context that was settled yesterday, having to restate a decision the model itself helped reach, is the contractor pattern showing through. Nothing is broken in the model; the continuity was simply never stored anywhere it could be read back.

§04 · statelessness is a design, not a defect

A boundary, not a malfunction.

Statelessness is not an error to be patched inside the model. It is a property of how the model is built and served: fixed weights, one request at a time, no write path back into the parameters. That property is also what makes a model predictable and shareable, since every call to it behaves the same way regardless of history.

Because the gap is a boundary rather than a bug, the way to close it is not to change the model but to give it something to read: a record that lives outside any one session and is written in a form a fresh call can pick up.

§05 · what can persist instead

A record outside the session.

If the model cannot hold the past, something else has to. The answer is a store the model reads at the start of a session and writes to as it works, so a later call inherits what an earlier one established. Substrate memory is one form of that store: a shared, typed graph that a fresh session reads to orient, rather than a transcript it must be re-fed.

A session that never saw the earlier work can still recognize it from the record rather than remember it, and a stateless model can boot into continuityin seconds by reading a briefing instead of replaying a history. The forgetting does not go away; it stops mattering, because the memory was never the model’s to keep.

§06 · common questions

Common questions.

Some assistants remember things between chats. How?

A feature layer outside the model does the remembering. It saves selected facts and re-inserts them into the context of a later conversation, so the model appears to recall them. The model itself is still stateless; the memory lives in the application around it, and what it holds is usually a short list of saved preferences rather than the reasoning of prior sessions.

Do larger context windows fix it?

They raise the ceiling within a single session, not across sessions. A larger window holds more at once, but it is still emptied when the session ends, and a long window can still lose track of detail in its middle. The cross-session gap is a separate problem from the size of any one window. See vs. a bigger context window.

Is forgetting even the right word?

Not precisely. Forgetting implies something was held and then lost. A stateless model never held the earlier session in the first place: each request is processed on its own. The everyday word for the experience is forgetting; the accurate description is that there was no memory to lose.

see also

Related entries.

introductionwhy LLMs forget
standard
the vinculum standard
The normative specification a conforming implementation is measured against.
read the standard →