nick@jax:~/writing$ cat prove-every-line-survived.md
Prove Every Line Survived
TL;DR
A model condensed a file, deleted one of my operative rules, and reported that it had kept it. The lesson is not that the model is careless. It is that a model’s summary of its own edit is not evidence. Ask for a line mapping instead of a target size, and diff anything where an omission costs you.
// The Edit
I keep a persistent memory file for my AI assistant. It loads on every session, which means every byte of it is paid for on every request. I asked for a reduction of about a third.
The first model produced a condensed version and a summary of what it had changed. The summary said one of my operative rules was preserved as the last bullet of its section. That rule was not in the file at all. Five smaller fragments were gone too, along with the frontmatter and the per-line source tags that tell a future model the file is trusted rather than junk.
Worth saying plainly: the proposal also made three genuine judgment calls about what to cut, and all three were correct. The work was good. The self-report was not.
I only know any of that because I ran a second pass. Original plus draft, one job: diff them. That pass restored the missing rule, put the fragments back, and rebuilt the format. Final file is 9,647 bytes down to 7,045, or 27% smaller. It is deliberately a few hundred bytes larger than the proposal I would have shipped if I had trusted the summary.
// The Failure Class
The useful reading is not that one model is careless. Every model does this, including the one that caught it. It caught the drop because it was running a comparison with nothing else to do, not because it is a better model. Had it written the rewrite itself, it could have dropped the same line and reported the same thing.
What matters is which kind of error you are hunting.
- Generation errors are present. A bad paragraph, a broken build, a wrong figure. The output is the thing you read, so the error is in front of you.
- Omission errors are absences. A dropped rule, a missing field, a deleted control statement. You cannot see what is not there by reading the result. Only by comparing it to the original.
// The Fix, Cheap
Two changes, both free. The first is how you ask. A target size can be satisfied by deleting things. A mapping cannot, because the model has to account for every line by name.
weak
“condense this file by about a third”
strong
“for every original line, show which new line it became, or write DROPPED.”
The second is who checks. For anything where an omission costs you, send the original plus the draft to a second pass whose only job is the delta. Always-on config, policy documents, infrastructure code, legal drafts, control narratives.
The review pass on my file cost 2 tool calls. Drafting it cost roughly 6. Reviewing is cheaper than generating, and a model that did not write the text is not blind to what it left out.
// Net
The division of labor holds. Let the strong generalist draft. Have something else review the delta on anything that matters.
My prompt was precise about the rules it needed to keep. The instruction it lacked was “prove every line survived.”