Baby Steps with Agents

I recently built a podcast serving and generation system for Commoncog Case Library while experimenting with coding agent workflows. Here is a report of how I built a moderately large system on top of our existing system using agents! Note that this was done in April - May 2026, and was representative of my approach then. I suppose it’s a testament to how quickly models and harnesses improve that my current development process today is wildly different compared to what it was when I built this system.

The codebase had been written mostly by humans on top of a Go framework. We had some tests, but there was not enough coverage. Given that the system will be fairly complex and I want the agents to write tests for the podcast system, I started working on the tests.

Testing with an ORM

Our tests do not use a mock database. Most of them write state into an in-memory SQLite and asserts by reading from the database. The Go framework’s ORM makes it easy to reuse the same code that uses SQLite in tests to use PostgreSQL in production.

Running a test against an in-memory SQLite has a few benefits. It is significantly faster compared to running them against Postgres, which would lead to a shorter iteration time for the agents to check its own work too. Using in-memory SQLite also meant that we could run many tests in parallel without bothering with database setup and teardown. In theory, you could set up many unique Postgres schemas, but in-memory SQLite is simpler.

Some behaviours, such as tests that exercise concurrent writes, are not supported by SQLite. So we do have some Postgres specific tests for these cases. But by running most of our tests against SQLite, we keep our tests fast.

Planning

Once I was happy with our test setup, I started planning our features by talking to an LLM. Most of these planning sessions ran on GPT-5.4 x-high. The output of a planning stage is a structured plan file, which we’ll get into in a bit.

I don’t plan all my features, but telling an agent to “create a podcast generation system, make no mistakes” is not going to work as there are just too many assumptions the agent needs to make correctly.

I plan in phases. If the feature is big and there are a lot of design decisions to make, I first give the planning agent a high level intent with extra instruction to ask me questions and clarify with me. This usually gets us to agree on the big picture scope. Then I ask the agent to write out what we agreed on as a plan file. This is usually a very lossy and high-level plan.

After that, I use the grill-me prompt template (originally from Matt Pocock, slightly modified) to reach a shared understanding. The prompt is really great, although it can easily lead to exhaustion due to making too many decisions at once.

Planning has now been the bulk of the work. The back-and-forth loop with the planning agent is done in a few stages.

The first phase focuses on user scenarios (given-when-thens). Locking in intents via grill-me is exhausting but is useful to prevent any surprises if/when you look at the implementation later. It is also cheaper and easier to change your mind during planning, before a single line of code is written.

The second phase focuses on structs, interfaces, functions and files to change. Surely the interfaces and functions you discuss with the planning agent might not always be the best interface, but aligning this with the agent is a good way to build shared understanding.

The third phase focuses on code reuse, test coverage and work splitting.

Why ask the agent to figure out which functions can be generalized / reused during planning time? As I’ve read more generated code, the current (as of April 2026) generation of agents’ instinct when seeing similar code snippets is to duplicate or wrap existing functions instead of generalizing them. Generalizing a function is harder because it needs to first check that all callers behavior remains the same after the change. But the DRY (Don’t Repeat Yourself) principle is not only useful to reduce cognitive load for humans, but also to prevent future codepath drift.

Left alone, these duplications will add up in your codebase, and signal to future agents that they should be treated as separate codepaths, which might then drift from each other. In either case, you’ll eventually need to do a refactor, which is much more painful because there is now a long gap between when the intent was first introduced and the refactor time. The cheapest time to fix is while the intent is still fresh. This is also known in quality engineering circles as the ‘Right First Time’ principle.

Maybe this third step is no longer needed by agents of the future as they are increasingly post-trained on this issue, but until then, finding code to reuse is a highly useful part of the planning phase.

The planning agent is also responsible for identifying high-leverage multi-step integration flows. An example might include updating content via the editor, asserting that it shows up in the presentation layer, and asserting that a podcast generation job is enqueued. Left alone, agents will happily create tons of tests that exercise local behaviors. But implementation details are prone to modification and have the shortest lifespan, which means tests that exercise local behaviors will also change often.

Integration tests are high leverage because it asserts behavior only at the edges — which means it remains useful for longer. As you might expect, integration tests exercise all parts of the codepath from the views layer to the controller layer up to the database, which means that these tests are closer to how users actually use the system. These test cases only test specific inputs, which means it cannot fully guarantee behavior correctness. What they do encode, though, is my expectation of what should happen, and that is useful for both future self and future agents. When these tests eventually change, it represents a fundamental behavior change rather than accidental implementation detail.

I also asked the planning agent to do commit-sized work splitting. At first the agent splits the work by layer. But as I accumulate plan files in the codebase, the planning agent is able to pick up my preference of splitting work by feature and to separate the refactoring step (most of the time this means no test changes) from the step that introduces new behavior. Including commit-sized steps in the plan makes it easier when I read the generated code later.

Sometimes, even after all this work, the commits might feel slightly too big. In such cases, I ask the agent to take some commit-sized steps, and create another plan file for that range of steps.

Depending on the plan complexity, I might start a new session to review the plan using the grill-me prompt template. This could sometimes uncover edgecases or behaviors that are contradictory with each other or are still vague in details, so it is good to do this as changing assumptions in a plan file is cheaper than after code and tests are written and assumptions are baked in.

Implementation

Experimenting with different Implementation Agents

I experimented with a few different implementation agents. The motivation is mainly cost. Implementation agents spend more output tokens than a planning or a review agent, so it is useful to try out other alternatives. (I still did all my planning exclusively with frontier models). I first used Cursor Composer 2 (posttrained on Kimi K2.5) via Cursor’s native harness, then Kimi K2.6, then Deepseek Flash V4, both on the pi.dev harness.

I was not happy with the result of Cursor Composer 2. Although it might be attributed to a less mature codebase state at the time I started using this model, I also found it worse at following instructions (both in AGENTS.md and when I issued steering instructions) and much too eager to write code before gathering enough context.

In the middle of my implementation, Kimi K2.6 was released. I started using it exclusively with pi. I found Kimi K2.6 to be better than Composer 2; it reliably got the job done. Based on anecdotal experience, it spends more time gathering context compared to GPT-5.4 at much lower token prices. However, it can be slower than GPT-5.4, and costs end up being comparable to GPT-5.4 for certain tasks due to the sheer number of tokens spent. I did not run any scientific evals but based on the vibes alone, if — or when — OpenAI decides to increase token prices, I know that an open-weight model can easily be swapped in its place.

Finally Deepseek V4 Flash was released, which came with absurdly cheap prices. I didn’t expect it to perform as well as Kimi K2.6, but surprisingly it did a pretty good job. It is sometimes less amenable to steering but as the codebase became more mature, I think there was barely a noticeable difference between it and Kimi K2.6.

Details

I experimented with how to use the implementation agents as well. It took some time for me to gain confidence to let a long running agent implement a full plan (committing multiple commit-sized steps without me skimming the code / making refactors in between each step). I currently believe that this is only possible in an AI-ready codebase. Without it, a few bad patterns will grow exponentially in your codebase such that the cost of refactoring becomes much too high.

A lot of the work in making a pre-AI codebase AI-ready is noticing what the agents repeatedly get wrong, and then fixing that. Noticing involves watching the thinking traces and the edits the agent made and seeing where it gets confused. When that happens, I added documentation to reduce similar confusion in the future.

The fix may also look like adding an instruction in AGENTS.md or CODESTYLE.md. Or refactoring to undo a bad pattern that the implementation agent introduces. I find that the agent can pick up bad patterns in your codebase and amplify it, so whenever you see one, it is very important to fix things before letting another agent loose.

Noticing also involves watching for when the agent adds unnecessary defensive checks in implementation functions and then undoing them. Unnecessary defensive checks in implementation functions are bad because it hides business logic bugs and turns them into runtime errors. You also lose the valuable signals you get when you have assertions in your code that an implementation function only handles certain types of valid inputs and would crash otherwise. During this implementation phase, I also maintained an INVARIANTS markdown file, which we’ll get back to.

On a similar note, noticing also involves watching when agents are swallowing or outright ignoring errors. These must be immediately corrected, as swallowing errors could lead to inconsistent state. Once again, this rhymes with what engineering has known for a long time which is to flag errors early.

Other fixes I’ve needed to make when the codebase was less mature is to ask the agents not to create impossible scenarios in generated tests (specifically scenarios that violate the INVARIANTS.md file). As tests also serve as documentation of intent, testing impossible scenarios obscures that intent and makes impossible situations look “possible”. Implementation agents are then misled, and write code with the assumption that such situations need to be handled.

INVARIANTS.md

I’ve mentioned the INVARIANTS file in passing. This file encodes, in human language, what the intent of the system should be and how the different subsystems are meant to interact.

A codebase is built around invariants. These invariants are also represented as tests, but maintaining invariants as plaintext reduces the need for subsequent agents to explore the codebase to (re)figure out the same invariants.

Test files can also grow large. When intent is buried across thousand lines of tests and a mix of localised unit tests and integration tests, it is annoying for both humans and agents to discover — and repeatedly rediscover — such intent in the future. With implementation agents, it also wastes valuable context budget. Keeping an INVARIANTS.md file helps the agent understand what is intended, what is possible, and what is not possible in a compressed manner.

I hooked up INVARIANTS.md by mentioning it in AGENTS.md so that it gets read when the implementation agent works on a podcast-related feature. (Some agents are not that effective at following this link, but as it is just a file, it is easy to mention the file at the beginning of an implementation session to load it into context)

Other than an INVARIANTS.md, I also maintained a CODESTYLE.md which encodes our codestyle preferences. Similar to INVARIANTS.md, this file is referenced by AGENTS.md and read by both the implementation agent and the review agent.

Actively managing context

Despite the direction that some agent harnesses (ahem Amp) are taking — which is to let the model manage its own context — I’ve been actively pruning context before each commit-sized step. This is easy in pi due to /tree . I think it is non-controversial that a big part of working with the current generation of LLMs is being ruthless about what is stuffed into their context window.

A typical session looks like this:

Prompt “Read plan, AGENTS.md, INVARIANTS.md, CODESTYLE.md”
Prompt “Implement commit 1 of the plan”
Skim resulting code
Make minor refactor and renames
Prompt “Please commit”
Rollback context to the state before step 2 using pi’s /tree
Prompt “We have implemented commit 1. Let’s implement commit 2”
Repeat step 3-5 until all stages in the plan are implemented.

Naming functions

In Go, the convention is usually to use fairly short function names, but with agents I think it is worth breaking that convention a bit. Slightly longer names are actually useful, because the agents search code with grep. It is better for them to see only the relevant things. Longer names are fine anyway, because you are not the one typing them all out every time. They lead to fewer recall turns and more relevant search results.

Single-use and single-line helper functions

This is an issue that gets on my nerves at times, especially as it doesn’t get better over time. That said, it’s easy enough to spot when you skim AI-generated diffs.

I am of the opinion that single-use functions should just be inlined to reduce premature abstraction. Single-line helper functions should also just be flattened. Why? Similar to the previous point, this reduces the number of greps the agent has to do and helps make use of the context window more efficiently. It is also because you don’t want this pattern to get amplified across the codebase.

Good News

The good news is that the current generation of agents are actually pretty good at picking up patterns.

After a few examples of batch loading patterns from the database to avoid N+1 queries, the planning model correctly identifies that the implementation should make use of this pattern when implementing batch actions.

Over time, I’ve also found that I needed to remind the agents less and less to not set up tests with impossible scenarios.

Some patterns are still prevalent, but I think as our codebase matures, working with agents should become easier over time.

Review

After an implementation agent completes a plan (usually 6-12 commit-sized steps), I ask GPT-5.4 x-high via Amp to review, feed the reviews back to the implementation agent, and then ask the review agent to verify the fix.

A few things that I specifically ask the review agent to look out for — aside from correctness — are test coverage gaps, deviation from CODESTYLE.md, potential simplification and insufficient code reuse. Even though the planning agent has thought about this and mentioned it in the plan, sometimes the implementation agent missed it or we discover a better reuse / simplification pattern.

Tweaking my harness

I spent quite a bit of time tweaking my pi harness. For instance, I gave it the ability to access the browser and verify that the new features are shown on the webpage as expected.

I believe this agent-polishing work will become a staple of coding with agents. Like with using tests, exposing agents to more feedback sources makes them more effective.

The key is to notice when agents go off-the-rails or are confused and then build them more sources of feedback.

Conclusion

People have written about Dark Factory patterns for a while now (Sam Schillace’s account), but have alluded to the huge amount of setup to get there.

Building one of the more advanced coding systems I’m talking about here takes a while to show benefits. It’s been something like 6 months of working on Amplifier and it’s only just now starting to be useful. (footnote 1)

Nearing the end of the podcast feature implementation, I made a change that altered a fundamental invariant. Unsurprisingly, pointing the agents at that change mostly works now. The code they generate also has much less slop than when I started. I think I now have a glimpse of what a Dark Factory might look like — though my work is nowhere near that level of automation. But I can’t wait to see what the future brings.

Published on May 20, 2026.