A Practical Guide, Part 4 of 15: Context and Memory

Host: Which leaves an awkward question. Every tool result lands somewhere. Today we find out where, and what it costs.

Host: Picture an agent three hours into tidying a messy codebase. It's read forty files, run the tests a dozen times, and it starts confidently repeating a fix it already tried. Nothing broke. It just lost the thread.

Co-host: I have watched that exact thing happen. It's like a colleague who's had too much coffee and not enough sleep.

Host: To see why, start with what context is. Anthropic defines it as the full set of tokens included when the model generates its answer.

Co-host: Quick definition: a token is a chunk of text, roughly a word or part of a word, which is the unit a model reads and writes. The context window is the maximum number of tokens it can look at in one go.

Co-host: So the loop keeps feeding the context, and nothing leaves unless someone takes it out.

Host: Anthropic's analogy is human working memory: the model has a finite attention budget, and every new token spends a little of it.

Host: So the practical shift is this: stop thinking of the context window as a bucket you fill, and start treating it as a budget you spend.

Co-host: Let me steelman the bucket view, though. Context windows have got huge. Why not just throw everything in and let the model sort it out?

Host: Because there's evidence that more tokens aren't free. Anthropic says that as the context grows, the model's ability to accurately recall what's in it declines. They call it context rot.

Co-host: Great name. Sounds like something you'd find at the back of the fridge.

Host: And position matters too. A research study called Lost in the Middle found models often do best when the relevant information sits at the start or the end of the input, and noticeably worse when it's buried in the middle of a long context.

Co-host: Which is how I read terms and conditions, honestly.

Host: Worth keeping the sources apart, though. Lost in the Middle is a measured study. Context rot is Anthropic's description of an effect, from a company that sells models with big windows, which arguably makes the admission more credible, not less.

Host: Their guiding rule follows naturally: find the smallest set of high-signal tokens that gives you the best chance of the outcome you want.

Host: In practice: before adding anything to an agent's context, ask whether it earns its place, and put what matters most where the model is most likely to see it.

Co-host: Okay, let's go back to our codebase agent. Walk me through how you'd keep it lean during one long session.

Host: Step one, just-in-time retrieval. Instead of loading forty files up front, the agent keeps lightweight identifiers, things like file paths or links, and only opens a file when it actually needs it.

Co-host: Like a librarian who knows the shelf numbers instead of carrying every book around. The trade-off is extra tool calls, and if the agent guesses wrong about what to open, it never sees the thing it needed.

Host: Step two, compaction. When the context gets full, you summarise the conversation so far and carry on from the summary.

Co-host: And this is where it bites. A summary is a judgment call. The one tiny detail that turns out to matter in an hour, say a test that failed once for an odd reason, is exactly what gets cut.

Host: Step three, structured notes. The agent keeps a running file outside the context, a to-do list, decisions made, things tried and failed, and rereads it when needed.

Co-host: That alone would have stopped our agent repeating its fix. It's the sticky note on the monitor.

Host: Step four, sub-agents. You hand a focused job to a separate agent with its own clean context, it does the messy exploring, and it returns only a condensed summary.

Co-host: The cost there is coordination. You're trusting the sub-agent's summary, and you pay for more total tokens across all of them.

Host: So the rule of thumb: retrieve on demand, compact carefully, write things down, and delegate the noisy work. Each one trades some risk of losing detail for a cleaner budget.

Co-host: This is starting to sound like how computers manage memory.

Host: That's precisely the idea behind a research system called MemGPT. It proposes virtual context management, borrowing the memory hierarchies of operating systems, and moves data between fast and slow memory to give the appearance of far more memory than the window holds.

Co-host: So the context window is the fast, expensive desk space, and outside storage is the slow filing cabinet, and the system shuffles things between them.

Host: Exactly. A second line of work came at it differently. Generative Agents gave a language model a complete record of its experiences written in natural language, had it synthesise those memories into higher-level reflections, and retrieve them dynamically to plan behaviour.

Co-host: Memory stream, reflection, retrieval. Did all three parts actually matter, or is that just a neat diagram?

Host: They tested that by removing pieces. Observation, planning and reflection each contributed critically to how believable the agents' behaviour was.

Co-host: Believable is a slightly soft measure, though. Being convincing isn't the same as being correct.

Host: Fair caveat. The lesson that carries over is design, not a score: decide what lives in fast memory, what goes to slow storage, and when raw memories get distilled into something more useful.

Host: Now imagine our agent comes back tomorrow. Everything in its context is gone. What does it remember?

Co-host: Nothing, unless you built something for that.

Host: One concrete answer is Claude's memory tool. It lets an agent store and retrieve information across conversations in a directory of memory files that persist between sessions.

Co-host: And as an engineer, the detail I care about: it runs client-side. Claude asks for file operations, and the developer's own application carries them out on storage the developer controls.

Host: In practice that pairs with a simple habit: check your memory directory before starting work, and write progress there as you go, so tomorrow's session can pick up the thread.

Co-host: Other platforms offer managed memory stores instead, where a service extracts and keeps facts for you. Less to build, less control.

Host: And research is pushing further. A paper submitted on 27 October 2026 proposes Proactive Memory, which organises an agent's history into recent events, trajectories over time, and behavioural regularities.

Co-host: Regularities sound like where you'd get superstitions. The agent decides every Tuesday is a disaster.

Host: They try to guard against that. Trajectory facts are checked against the original events, and regularities are tested by replaying history before being matched to what's happening now.

Host: They also built a benchmark that follows agents over time, across three application areas, processing separate event streams in order.

Host: On it, Proactive Memory reached an overall F1 of 52.6 percent, a 21.4 percent relative improvement over the strongest of five outside baselines.

Host: Agreed. And there are open questions nobody has fully answered: what if a malicious document plants a false memory, who can read those files, and how do you stop stale memories quietly misleading the agent?

Host: So if your agent must return later, treat memory like any stored data: decide what's worth keeping, verify it, control access, and expire what's out of date.

Host: The one takeaway: the context window is a budget, not a bucket. Spend it on the smallest set of high-signal tokens, and push everything else into notes, retrieval or memory you can check.