The more reasoning I handed the AI, the more it made up

A recent project at work was a big one: migrate a backlog of old .NET Framework applications onto .NET 10, dozens of separate customer codebases, each its own pile of years-old code, all pulled into a single modern repo. I built a set of Claude Code skills to do the heavy lifting. Scan a legacy codebase, compare it against the new shared baseline, work out what is genuinely custom and what is just old framework noise, scaffold the new structure, and hand a developer a report to review.

The early versions were almost good. Almost is the word that cost me weeks.

Almost

Here is what kept happening. The instructions for a real migration are long. There are conventions, file layouts, things that must never be copied over, edge cases for every kind of customization. When the model had all of that in its head at once and had to reason its way through a few hundred files, it drifted. It would skip a step it had been told twice to do. It would write a vague description where I asked for a specific one. Worst of all, it would invent things: a column that was not in the database, a method that did not exist in the file it was supposedly reading. Confident, well-formatted, and wrong.

And the more context it was carrying, the worse it got. The rules I cared about most got buried under everything else it was juggling, and somewhere past the middle of a long run they quietly stopped winning.

I tried to fix it the way everyone tries first. Better wording. More emphasis. Bigger, more careful prompts. ALL CAPS on the important rules. It helped a little and then it didn't, because I was treating a structural problem like a phrasing problem. You cannot prompt your way out of a model having too much to hold.

Take the reasoning away from it

The thing that finally worked was the opposite of what I had been doing. Instead of making the instructions better, I made the model's job smaller. Anywhere the work was actually mechanical, exact, and the same every time, I pulled it out of the model and gave it to a Python script. The model stayed in charge of the parts that need real judgment, and nothing else.

The clearest example was a step that asks the model to describe what changed in each file. One agent, the whole list of diffs, write a short specific description for each. It failed the same way every time. After about fifty files it lost the thread and started writing "updates code logic" for everything. Sometimes it skipped entries with no error at all, just a quiet gap where a description should be.

So I split it up, and the split is mostly code, not prompt:

• A script handles the obvious ones with no AI at all. A renamed project file, a binary, a file that is identical on both sides. Those never reach the model.
• The rest get chopped into small batches, about fifteen at a time, and each batch goes to a fresh agent that sees only those fifteen files and nothing else. No long context to drift through.
• A second script checks every description that comes back, and it has no opinion. Too short, gets rejected. Matches a list of filler phrases, gets rejected. Just repeats the file name, gets rejected.

# the validator does not reason, it just refuses filler
BANNED = ["updates code logic", "minor change", "modifies functionality"]
if len(desc) < 40 or any(b in desc.lower() for b in BANNED):
    send_back_for_rewrite(entry)

The model still writes every description. It just never holds five hundred of them at once, and nothing it writes gets through without passing a check that cannot be charmed.

A script cannot be talked out of a finding

The final check that a migration is actually complete is a plain Python script. Do the required folders exist, did the old config get cleaned out, does the report have all its sections. The model runs the script and fixes whatever it flags. It does not do the checks itself.

That distinction matters more than it looks. Ask a model to verify its own work and on a long enough run it will decide a missing file is probably fine, or that it remembers creating something it never did. A script does not get tired and does not round things off. It either finds the file or it doesn't.

I do the same thing with the report the developer reads at the end. A script generates it, not the model writing HTML freehand. Let the model build the page each time and you get a slightly different layout on every run. A script gives you the same report shape every single time, which is the whole point when a developer is comparing forty of them.

And then the database said hello

For a while I thought of this as a coding-agent trick. Then I hit the exact same wall on the database side, and the fix turned out to be the same shape with a completely different tool.

Part of the upgrade is comparing a customer's old database to the new one. Schemas, data types, the lot. This is precisely the kind of job a model will get almost right and hand you with full confidence. It drops a column. It invents a constraint that sounds plausible. It summarizes when you needed the actual list. Almost right, with a database, is just wrong that hasn't surfaced yet.

So I did not have it read the schemas. I had it run SqlPackage, the SQL Server schema-compare tool, plus a set of plain SQL queries, and let those do the actual comparison. They are built for exactly this and they are exact every time. The model's job shrank to running the tool, reading the structured result it produces, and turning that into the list of decisions a developer signs off on. The comparison itself never happens inside the model.

The interesting part is what the right tool turned out to be. The second time around it was not Python. It was SQL and a schema-diff engine that already existed. And that is the real lesson, the one bigger than "write a script." The point is to find the deterministic thing that already does the exact part, whatever it happens to be, and build the model around it. For the code it was Python and a pile of regex. For the database it was the database's own tooling. The model orchestrates. It does not do the part that has to be right.

What I actually believe now

A prompt is a suggestion. It leans the model in a direction, it does not bind it, and the longer and more complex the run, the less that lean holds. For anything where "usually right" is good enough, a suggestion is all you need and it works great.

A migration is not a usually. A skipped file or an invented column is a wrong answer wearing a confident face, and you might not catch it until production does. So the line I keep landing on is this: if a piece of the job is mechanical and you can check it without judgment, it does not belong in the model's head. Write the script, or better, go find the tool that already does it, and give the model only the part that genuinely needs a mind.

The smaller you make its job, the less room it has to wander off and make something up. Most of my work lately has been figuring out where that line sits, and moving everything on the wrong side of it into code.

Thanks for reading. If you have hit the same wall and solved it some other way, I would like to hear it.