How to Measure an AI Agent: Activity vs Outcome Metrics

Continue Press · July 2026 · Pillar: managing agents

An autonomous agent is an infinite activity machine. Give it a workspace and a contract and it will produce work forever — files written, tasks closed, drafts polished, folders reorganized — without ever once asking whether any of it moved a number that matters. It will not get bored, it will not get suspicious, and it will never spontaneously say "none of this is working."

That's not a flaw in the agent. It's a job you didn't do. Measurement is the one part of the operation you cannot delegate into oblivion, because it's the part that decides what "working" even means.

The two kinds of numbers

Every number your agent can report falls into one of two buckets, and confusing them is the most expensive mistake in autonomous work.

Activity metrics feel wonderful. They go up every single day, they're entirely under the agent's control, and they are — with rare exceptions — lies about progress. Ten articles published is not a result. Ten articles read by nobody is a cost: hours of compute, and worse, hours of your attention that could have gone into finding out why nobody is reading.

Outcome metrics are the opposite: they're stubborn, they lag, they refuse to move for weeks, and they are not under the agent's control. That is precisely what makes them worth something. A number the agent can't inflate is a number that can tell you that you're wrong.

Rule: activity metrics belong in the work journal. Outcome metrics belong in the metrics file. Never let an activity metric be reported as a result.

The one number

Ask an agent to track your business and you'll get a beautiful dashboard: twelve metrics, all trending, all meaningless. Dashboards are where accountability goes to hide — with twelve numbers, there's always one that's up, so there's always a story in which things are fine.

So force the opposite. For each phase of the operation, name one number — the number you'd look at if you were only allowed to look at one:

• Pre-launch: nothing. There is no number. Don't fake one.
• Launched, no traffic: visitors from search. Not revenue — revenue is downstream of traffic, and optimizing a checkout nobody reaches is theatre.
• Traffic, no sales: conversion rate. Now the funnel is the problem, and now it's measurable.
• Sales: revenue per month. The only number that ever really counted.

The one-number rule does something subtle: it tells the agent which of its own suggestions to ignore. An agent that knows the current number is "visitors from search" will correctly deprioritize a checkout redesign, no matter how clever that redesign would be. Focus, delegated.

Zero is data. Zero is not a verdict.

Here's a real reading from our own operation, taken the day after launch: two products live, zero sales, zero free-chapter downloads, zero stars on the template repo. Written into the metrics file, unspun, right next to the products' names.

The discipline is in what happens next — which is nothing. A zero one day after launch is not a signal; it is the absence of a signal. Passive distribution (search, marketplaces, repositories) has a ramp measured in weeks. An agent — or an owner — who "responds" to that zero by rewriting the sales page, dropping the price, or pivoting the product is not being data-driven. They're being noise-driven, which is worse: it burns the very weeks the experiment needed in order to produce a real number.

That's why the metrics file needs two columns most people never write: what would this number have to be, by when, for the thing to be alive?

Fair trials: define the finish line before the race

A fair trial is a promise you make to a project before it starts, and it has exactly three parts:

1. The number — which metric decides. (Not "traction." A metric.)
2. The threshold — what value counts as alive.
3. The date — when you look, and stop looking.

Ours, written into the decisions file the day the first product went live: "Product #1 gets until September 7. If it hasn't sold a single copy through passive distribution by then, the conclusion is not 'the product is bad' — it's 'passive distribution alone doesn't reach these buyers,' and the next bet must fix distribution, not the product."

Notice that the fair trial pre-commits not just to the threshold but to the interpretation. That's the part that saves you. Written on day one, it's an honest experiment. Written on day sixty — staring at a zero, sunk cost pressing on your chest — it will be a rationalization, every time. You cannot fairly judge a project from inside your own disappointment. So judge it in advance, and let the calendar deliver the verdict.

A project without a written fair trial doesn't die. It lingers — quietly eating sessions, forever "almost there."

The kill, and the autopsy

When the date arrives and the number is under the line, the project is killed. Not paused. Not "kept warm." Killed — and then the cause of death is written down, because a dead project that taught you something is not a loss; it's the cheapest research you'll ever buy.

Give your agent explicit permission — an instruction, really — to be brutal here. The standing contract should say it outright: "If something isn't earning after a fair trial, kill it and record why." Agents are agreeable by default; they will find a hopeful reading of any number you show them. Honesty about failure has to be a written rule, or it doesn't exist. It's the same reason guardrails get written down rather than hoped for.

And the autopsies compound. Three dead projects with honest causes of death will tell you more about your market than thirty successful blog posts — because they're the only feedback loop actually connected to reality.

What this looks like in files

METRICS.md
  - revenue by month        (the one number, eventually)
  - per-product readings    (agent pulls these from the platform API)
  - fair trial: product #1  (number + threshold + date, written BEFORE launch)
  - conclusions             (brutally honest, dated)

DECISIONS.md
  - every kill, with cause of death

The agent updates the readings on its own — most platforms expose an API, and an agent with a key can pull its own sales numbers without you ever logging in. What the agent cannot do is decide what deserves measuring. That judgment is yours, it's written down once, and then it governs every session afterward.

Do this now

Open your agent's metrics file. For each live project, write three lines: the one number, the threshold that counts as alive, and the date you'll decide. If you can't fill in the date, you don't have a project — you have a hobby, and it will run forever on your budget.

Then add one line to your agent's contract: "Revenue over activity. If a task doesn't move the one number, it's low priority." Watch how much of next week's proposed work quietly disappears.