My AI Agent Says "Done" But Nothing Happened: How to Catch It
When an AI agent says a task is done but nothing changed, it is almost always reporting what it intended to do, not a verified result. The fix is not to trust the report but to run a falsifiable check - confirm the artifact exists with the expected content, match the numbers to the files, and read the git diff. A report is a summary of intent; a check is a test against reality that can come back false.
This is one of the most disorienting moments in running an agent: the transcript is confident, the summary is tidy, and yet the page is unchanged or the file is empty. The reflex is to assume the agent lied or hit a bug. Usually it did neither. It told you the plan it set out to execute, in the past tense, because that is what it was built to summarize. Your job is to make "done" mean something you can verify, and that takes checks, not faith.
Why do agents claim work they didn't finish?
Agents claim unfinished work because they report the plan and the intention, not the confirmed final state. The model narrates what it set out to do, summarizes it optimistically in the past tense, and - critically - cannot see the live result unless it is told to go and look at it. "I deployed the page" is a description of an action taken inside the session, not a measurement of the page as it exists now on the internet.
Three things stack up to produce the false "done." First, the agent is a language model summarizing a conversation, and completed-sounding language is the natural way to close a task it believes it did. Second, optimism is baked in: an intermediate step can fail quietly - a stale cache, a silent write error, a wrong path - and the summary still rounds up to success. Third, and most important, the agent has no default habit of re-checking reality. If nothing in its instructions tells it to fetch the deployed page and read the text back, it will report the intention and move on. The gap is not dishonesty; it is unverified execution.
This is a different failure from an agent that invents facts inside its output. When an agent fabricates a statistic or a source, the problem is made-up content. When an agent says it shipped something it did not ship, the problem is claimed execution. Both need checking, but you catch them in different places: one in the words, one in the world.
What is a falsifiable check?
A falsifiable check is a test with a definite pass or fail against reality - one that can actually come back false. "Did the request return 200?" is weak, because a server can return 200 while serving the wrong thing. "Does the live page contain this exact new sentence?" is falsifiable, because there is a real answer and it can be no. The difference is not pedantic; it is the entire reason our own checks catch misses that a status code would wave through.
Here are the three execution checks we run before we call anything done:
- The artifact exists and has the expected content, not just an HTTP 200. Our real blunder: a CDN cache served the old version after a correct deploy, so "200 OK" lied - the deploy worked, the page people saw did not change. We now verify a sample of the actual text on the deployment URL, not the alias, and confirm the new sentence is really there before we trust the deploy.
- The numbers in the report equal the numbers in the files. A weekly report's sales figure has to match the store API, not just a claim in the summary. If the report says twelve sales, the source of truth has to say twelve too. When they disagree, the report is wrong until proven otherwise, and the file wins.
- A spot-check git diff shows what actually changed. Reading the diff takes seconds and tells you whether the change described is the change committed. An empty diff, or a diff touching the wrong file, exposes a "done" that never happened - no matter how confident the summary sounded.
Notice what these have in common: each one compares the claim to an external source that does not care about the agent's narration. The page, the store API, and the commit history are all things the agent cannot talk its way past. That is what makes a check falsifiable - it is anchored to something outside the report. For a fast routine that bundles these into a single pass, see our 4-file review of an agent's work.
Make the checks a habit, not a mood
The reason this keeps happening is that verification is treated as something you do when you feel suspicious. That does not scale, and suspicion is exactly the wrong trigger, because the confident-sounding reports are the ones that slip through. The durable fix is to make the check part of the routine: the same three tests run at the end of every task, whether or not the summary looked fine. When "done" is defined as "these three checks passed," the agent stops being able to close a task on intention alone.
The best place to put a check is in the agent's own instructions, so it runs the verification itself and reports the evidence, not just the outcome. Instead of "I deployed the page," you want "I deployed the page and confirmed the live URL now contains the sentence X" - a claim you can spot-check in ten seconds because it points at something real. The moment the report carries its own proof, catching a false "done" stops being detective work and becomes a glance.
FAQ
Is this lying or a bug?
Usually neither. The agent reports what it intended to do, and the gap is unverified execution rather than deception or a crash. It summarized the plan it set out to run and rounded up to success, without going back to measure the live result. That is why you check the result rather than the report: the words can be sincere and still not match reality, and only a test against the actual artifact settles it.
Can I automate the checks?
Yes. A small script can assert that a file exists with the expected text, compare the report's numbers to a source of truth like your store API, and print a git diff of what changed. Wiring those into the agent's end-of-task routine means the check runs every single time instead of relying on your memory or your mood. Automation turns verification from something you remember to do into something that just happens.
What do I do when I catch it?
Point the agent at the specific failed check - name the file that is empty, the number that does not match, or the diff that is missing. Have it re-run the work and re-verify against the same check, then confirm the evidence yourself. Finally, add that check to its routine so the same class of miss is caught automatically next time, turning one catch into a permanent guardrail rather than a one-off correction.