A Practical Guide, Part 11 of 15: Agents That Improve Themselves: Reflection, Feedback, Growing Their Own Skills

Host: And we framed automate versus supervise as a risk decision. Today we ask a slightly unnerving question. Can an agent get better on its own, and when should we believe it when it says it has?

Co-host: Which, for the record, is a question I also ask about myself after every code review.

Host: Start with that note-taking agent, because it's a real design. Reflexion agents improve by reflecting in plain language on the feedback from a task, then storing that reflective text in an episodic memory buffer. No weight updates at all.

Co-host: So mechanically, walk me through one cycle. What's computed, in what order?

Host: In general terms: the agent attempts the task, gets a signal back, say failing unit tests or an error. Then it's prompted to explain to itself what went wrong and what to do differently. That explanation goes into memory, and on the next attempt it's placed in the context alongside the task.

Host: Exactly that. The strongest version of the case is the headline result. Reflexion reports ninety one percent pass at one on the HumanEval coding benchmark, against eighty percent for what was then the state of the art, GPT-4.

Co-host: That's a big jump for, essentially, a diary. What's the catch from an engineering side?

Host: A few. Every reflection costs tokens and latency, and the memory eats context window. And the trick works best where the feedback is crisp. Coding has tests. A vague task gives the agent vague feedback, and you get confident notes about the wrong lesson.

Co-host: Garbage in, eloquent garbage out.

Host: Now a close cousin, Self-Refine. One language model plays all three parts: it writes a draft, gives feedback on it, then rewrites. No supervised data, no extra training, no reinforcement learning.

Co-host: Same model, three hats. And does that actually help, or is it the model admiring itself in a mirror?

Host: It helps, measurably. Across seven tasks, both humans and automatic metrics preferred the refined outputs, with an average improvement of about twenty percent absolute in task performance.

Co-host: Okay, but steelman then test. What does preferred actually mean there?

Host: That's the right worry. Preferred means better liked, or scoring higher on a metric. It does not mean correct. A refinement loop can polish a wrong answer until it reads beautifully. The model's critique can only catch errors the model can recognise.

Co-host: Which is the same blind spot that made the error in the first place. So if I'm building one of these, when do I stop iterating?

Host: In practice you want a stopping rule that isn't the model's opinion. A fixed number of rounds, a budget, or an external check passing. Otherwise you pay for round six when round two was already as good as it gets, and occasionally round six makes it worse.

Host: So what it means for you: verbal reflection and self-feedback are cheap, real improvements with no retraining, but they shine where there's an outside signal like tests, and you should cap the loop rather than trust it to know when it's done.

Co-host: Reflection helps on the next attempt. But what about something an agent learns once and reuses forever?

Host: Enter Minecraft. Voyager is an agent that combines three pieces: an automatic curriculum that proposes what to tackle next, a skill library of executable code that keeps growing, and an iterative prompting loop fed by environment feedback, execution errors and its own self-verification.

Co-host: I love that the skills are code. That's the bit that makes me sit up as an engineer.

Host: Why does code matter so much to you?

Co-host: Because code composes and it can be run. A note saying be careful near lava is advice. A function that mines iron safely is a tool you can call inside a bigger function. You get a library that compounds, and you can actually test each piece.

Host: And none of it touches the model. Voyager talks to GPT-4 purely through black box queries, so there's no fine-tuning of parameters anywhere.

Co-host: So a worked example, generally. The curriculum says, build a shelter. The agent writes a function, runs it, the game throws an error, that error goes back in the prompt, it rewrites, and once it judges the task done, the function gets saved for next time.

Host: That's the shape. And notice where it can fail. If the self-verification says done when it isn't, a broken skill goes into the library and gets reused. One bad brick, many walls.

Co-host: And libraries rot. Names collide, two skills do nearly the same thing, retrieval picks the wrong one.

Host: Which brings us to how this looks in practice today, outside a game. A common pattern is to package skills as folders: a short description up front, fuller instructions underneath, and scripts or reference files deeper still. The agent sees only the descriptions until a task needs one, then loads more.

Co-host: Progressive disclosure. Yeah, the context window is the scarce resource, so you show the menu, not the whole kitchen.

Host: Nicely put. And the obvious next step, which people are openly aiming at, is letting agents write, edit and evaluate their own skill folders. That's Voyager's idea moved into everyday tooling. I'd treat it as a direction, not a solved problem.

Co-host: Because the evaluate part is doing a lot of quiet work in that sentence.

Host: What it means for you: if you want learning that lasts, store it as small, testable, reusable units with short descriptions, load them on demand, and never let a skill into the library on the agent's word alone.

Co-host: So far the agent edits notes and tools. What's the radical version?

Host: The agent edits itself. The Darwin Gödel Machine repeatedly modifies its own code, and checks every change empirically by running it against coding benchmarks.

Co-host: Its own code meaning the agent scaffolding, the tools and prompts and logic around the model, rather than the model weights?

Host: That's how I'd read the agent's code, yes, and the key word is empirically. It doesn't try to prove a change is better. It tries it and measures. The results are striking: on SWE-bench it went from twenty percent to fifty, and on Polyglot from fourteen point two to thirty point seven.

Co-host: Those are big moves. Steelman, then poke. What's the strongest honest version?

Host: Strongest version: a system found improvements to its own design that its designers hadn't written, and an external benchmark, not its own opinion, confirmed them. That's precisely the independent check we keep asking for.

Co-host: And the poke?

Host: Optimise hard against a benchmark and you may get better at the benchmark rather than at the job. Every measure has gaps, and a search over your own code is very good at finding gaps. Plus it's expensive, lots of candidate versions, each run through a full evaluation.

Co-host: And safety. Code that rewrites its own code is the sentence that makes security teams put their coffee down.

Host: The researchers ran the experiments with precautions, including sandboxing and human oversight.

Host: So what it means for you: self-modification is a real research result, not science fiction, but it belongs in a sandbox, with a human gate on anything that ships, and you should read benchmark gains as gains on that benchmark.

Host: Now the thread running under everything today. Every loop we've discussed has a moment where someone decides, that was better. The uncomfortable question is who that someone is.

Co-host: And from what I've seen building these, when the agent grades its own homework, it's a generous teacher.

Host: That's my view too, and it's widely shared among practitioners: self-assessment tends to lean positive. The model that missed the bug is the same model asked whether there's a bug. It's like proofreading your own email. You read what you meant to write.

Co-host: So, concretely, how do I design around that? Give me the checklist.

Co-host: And notice the papers that did best already had that. Reflexion on code had tests. Voyager had the game throwing errors. The Darwin Gödel Machine had benchmarks.

Host: Yes, the external signal was carrying a lot of the weight. Where practice differs from the papers is that most real tasks don't come with a tidy benchmark. Writing a support reply or a contract summary has no unit test, and that's exactly where self-evaluation is least trustworthy.

Co-host: Open problem, then. How do you get honest feedback on fuzzy work without a human reading everything?

Host: Honestly, nobody has a clean answer. Sampling outputs for human review, checking specific properties you can verify, and logging every self-change so you can roll it back are the sensible hedges. And it's slightly funny that an AI-voiced show is warning you not to let AI mark its own work.

Co-host: We've been graded externally. By you, the listener. Please be kind.

Host: What it means for you: trust self-evaluation only as far as an outside check confirms it, and the fuzzier the task, the more human review you budget for.

Host: So the one takeaway: agents can improve without retraining, through reflection, self-feedback, skill libraries and even rewriting their own code, but the improvement is only as real as the independent test that confirms it.

Host: This was part of our series Agentic AI: A Practical Guide. The show notes list every source, and this episode was made with AI voices. Follow AI Shadow for the rest of the series. Goodbye for now.