Your Agent Work Is Already a Dataset

Last month our fleet ran 1,731 agent sessions and spent $17,816. I can tell you the cost of any session to the penny, which model it used, and what directory it ran in. Here is what I can’t tell you from any of that telemetry: what the work was for.

Every question worth asking about agent spend is shaped that way:

• Why did we spend $18k on agent work last month?
• What work came out of that architecture decision?
• What did this migration actually cost?
• Did the decision we approved turn into the implementation we expected?

Those are questions about intent. A session record knows where work happened, who ran it, and what it cost. Why it exists lives in someone’s head until someone writes it down. More often, nobody does.

Most agent observability asks whether the agent did the task well. The harder question is why the task existed. Capture that one field and your session history becomes queryable by intent: work you can group, price, and compare by what it served.

The blast radius took another round of archaeology

Three weeks ago I published the blast radius of one architecture decision: a $71.51 decision phase that grew into $4,633.23 of downstream work, a 65× amplification floor. Knowing that number changed how we review decisions, so I wanted it fresh for September, because the decision was still generating work.

The September number exists: $4,728.15, which is 26.5% of the month’s fleet spend. Getting it hurt. The original analysis chained sessions to the decision by hand, prompt by prompt. The update meant doing that again, plus hand-updating a tracking signature for six new repo names before it caught the thread’s biggest session. You need the blast radius. You can’t keep paying for it in archaeology.

Five ways to attribute AI agent spend, scored on 1,731 real sessions

So I scored the cheap reconstructions, the ones that use data we already collect. Four failed. The fifth gets close, and the way it falls short is the point.

Search for the decision’s name. cassette in titles and paths returns 338 sessions and $2,334.37. That’s 49.4% of the thread’s cost, and it misses the most expensive session in the entire census: $2,130 over 119 turns, title and path never say the word.

Broaden the keyword. tapes catches that session and returns 113 matches, 90 of them other people’s work. Tighten the search and you lose work. Widen it and you lose meaning.

Group by identity. 81% of our census runs under one auth subject, the eval fleet’s, and a human shares it now. Who ran a session stopped predicting what it was for.

Trust the time window. The decision happened in July. A thirty-day window had forgotten it by September while the spend kept arriving.

Infer it from the content. The strongest option, so I gave it its best case. I didn’t run a model; I scored every prompt, title, and path with perfect knowledge of the decision’s vocabulary. It recovers 96.7% of the cost at 92.2% precision, and it still can’t tell this decision from its neighbors, because which work belongs to which decision is something only the organization knows. Attribution also needs a ledger, and this is a probability; FinOps already uses declared cost tags because some attribution has to be explicit. The $158.20 it missed never says cassette anywhere. A smarter model can catch that work once it knows tapesctl belongs to this decision. But that mapping still has to come from somewhere. Someone has to declare the relationship.

Try it yourself. The interactive below drops you into the same census with a search box and live scoring: what you catch, what you falsely catch, and what you miss. My best keyword recovered 49.4%. See if you can beat it, then press the label button.

(Full-page version: find the thread.)

So we added a label

A commit has a SHA. A ticket has a key. A request carries a trace ID through every hop. The decision that causes all of them usually gets nothing. So the fix is small:

paperctl label create decision:cassettes
paperctl label add <session-id> decision:cassettes
paperctl sessions list --label decision:cassettes

The obvious objection is that this starts manually. It does. The manual part is the declaration. A request gets a trace ID because the system watched it arrive. No system watches a decision arrive; it happens in a meeting, a doc, a thread. What a session is for is a fact about your organization, and it isn’t in the trace. So a person who knows says so, once, and the result is a record instead of a guess. Everything after that first attachment should be automatic, and today it isn’t: paperctl won’t propagate a label yet, but labels attach after the fact, one session or a scripted batch at a time. Minutes in the same week. Archaeology a month later.

What you attach determines what the label is worth. repo:console says where work happened, and telemetry mostly knows that already. decision:cassettes says why the work exists, which the session trace doesn’t reliably preserve on its own. The same pattern covers migration:*, incident:*, experiment:*, customer:*. Attach the intent while everyone still knows it.

One honest note: our census has no labels column. Every grouping in this post was reconstructed after the fact, which is exactly the workflow labels retire.

A few months of labels becomes a dataset indexed by why

In week one a label is a filter. Keep going and each label becomes a join key:

decision:cassettes → the sessions that served it → the people and repos they touched → the cost → the rework → the outcome.

With the label attached, I can ask about the decision instead of searching for words I hope still travel with it. Every question at the top of this post becomes a query. The receipts are already in our own thread: an urgent production fix and a scrapped $42 subsystem that no project-keyed accounting would ever attribute back. With labels, each would have been one filter away instead of a re-pull. And the last question on the list, whether the decision we approved became the implementation we expected, turns into a diff: the RFC on one side, the label’s record on the other.

The labeled set is also an input you chose on purpose. Export it and you have the execution record attached to that decision. Tag your best sessions skill:cassettes-docs along the way and you can generate a reusable skill from them when the thread winds down. The mechanics live in two short guides: build a dataset from your sessions with labels and create a skill from labeled sessions.

Give the decision an ID

Git indexed engineering by code changes. Tracing indexed it by requests. Session records with labels index it by why the organization did the work, and that’s the index every question above needs.

Pick the highest-fan-out decision your team has in flight and give it a label this week; it costs minutes. A month from now, pull everything carrying it and set it next to what you approved. The gap between those two records is where your next architecture conversation should start.