Definition
AI coding session analysis is the practice of reading recorded AI agent session data (prompts, tool calls, errors, and outcomes) to identify patterns, extract reusable procedures, debug failures, and improve the reliability and efficiency of future agent runs.
The main ways to capture and analyze AI coding session data fall into a handful of architectural categories: raw session history, application-level instrumentation, telemetry-based tracing, proxy or provider-boundary capture, and purpose-built session analysis. They differ mainly in what they can observe and in what they let you do with what they find. This page covers why the practice exists, what it is not, how the approaches compare, and what kinds of improvements analysis can produce.
Why AI coding session analysis exists
Agents rediscover solutions. An engineer’s agent works out a fix for a flaky integration setup on Tuesday; a teammate’s agent works out the same fix from scratch on Thursday. The knowledge existed; it just lived in a session record nobody read. And failures repeat the same way: the error path one session already diagnosed shows up again in the next run, undiagnosed.
Session analysis is the practice that closes that gap. The end product is not a dashboard metric. It is a change to future runs: a pattern identified, a failure understood, a procedure lifted out of one session so the next one does not have to rediscover it. Without analysis, session data is exhaust: generated at cost, then discarded.
The practice also corrects intuition. When we audited 500 of our own team’s sessions, the waste everyone expected to find (agents repeating the same calls) turned out to be marginal, and the costly patterns only surfaced by reading what the agents actually did.
In an audit of 500 team sessions, exact repeated tool calls were only about 4% of the total. The expensive patterns were quieter, and they surfaced only when the trajectories were read.
What session analysis is not
The term sits near several practices it should not be confused with:
- Not APM or observability dashboards. Latency, cost, and error-rate visibility (the territory of agent observability) tells you the system is healthy. Session analysis reads the content of the interaction to change how future runs behave.
- Not code review. Code review evaluates the artifact a session produced: the diff. Session analysis evaluates the trajectory that produced it: why the agent made the calls it did, and what that implies for the next run.
- Not model evals. Evals score a model’s behavior against benchmarks under controlled conditions. Session analysis reads real sessions doing real work, with all the mess that entails.
- Not token-count analytics. Usage metrics answer how much. Analysis answers why, and what to do differently.
How analyzing AI coding session data works
Understanding the tools requires understanding what the data is. A captured AI coding session is not a log file. It is a sequential record of an interaction:
| Prompts | The user instructions and system context at each turn, including tool results fed back to the model. |
|---|---|
| Model responses | The model outputs: reasoning, plans, code, and tool invocations. |
| Tool calls | The tools called, their arguments, and the values they returned, as recorded at the capture boundary. |
| Errors | Error messages, failed tool calls, and exception traces, including the model response that followed each failure. |
| Outcome | Whether the session successfully completed its task, and the final state of any artifacts it produced. |
| Metadata | Model version, timing data, token counts, provider routing, and other session-level facts. |
Not every approach captures all of these fields. The first question to ask of any approach is: what can it actually observe?
Raw or native session history
The simplest starting point: the records the tools already produce. Terminal transcripts, an agent’s own session history, exported conversations or session files.
What it does well: Zero infrastructure cost. Best for occasional manual inspection: reviewing a single session after a failure, understanding a specific decision, checking a recent run.
Where it falls short: Cross-session analysis is limited unless additional tooling is added. Hard to find a session from two weeks ago, harder to spot a pattern across an engineer’s full history. Produces no reusable artifacts on its own.
Application-level instrumentation
The agent or application is instrumented directly (through SDKs, hooks, callbacks, or tracing libraries) to emit events as it runs.
What it does well: Can capture rich, application-specific events, because the application explicitly emits them: internal state, custom milestones, domain-level annotations that no external observer could infer. Useful when a team controls the application and wants structured telemetry.
Where it falls short: Coverage depends on what is instrumented. Events nobody wired up are invisible, and each new tool or framework needs its own integration work.
Telemetry-based tracing
Agent and model activity is represented as structured traces, spans, events, and attributes, sent to an observability backend.
What it does well: Useful for debugging, operational analysis, aggregation, evaluation pipelines, and integrating AI activity with the observability systems a team already runs: AI traffic appears alongside the rest of the distributed traces.
Where it falls short: Session-level usefulness depends on the instrumentation and the data model. Spans typically capture inputs and outputs per call; the sequential decision trail across many turns requires additional modeling on top.
Proxy or provider-boundary capture
An intermediary sits between the agent and the model provider, and records the traffic that crosses it.
What it does well: Can reduce the need to instrument every model call individually: traffic routed through the intermediary is recorded regardless of which tool or framework produced it.
Where it falls short: Sees only what crosses that boundary. Local shell operations, filesystem activity, MCP calls, UI state, and other agent-side events may not be directly visible unless they are represented in the provider traffic.
Purpose-built session analysis
The coding-agent session, the trajectory itself, is the primary object rather than a byproduct of logging or tracing.
What it does well: May add capabilities such as session search, cross-session pattern analysis, outcome analysis, eval generation, and reusable artifact extraction, because the data model is built around trajectories rather than events.
Where it falls short: Capabilities vary by implementation: no universal claims hold about search depth or artifact generation. And a purpose-built layer still depends on one of the capture positions above for its raw material, inheriting that position’s boundary.
The question is not which approach has the best dashboards. It is whether it produces something you can act on.
| Approach | Capture boundary | Setup | Multi-turn trajectory | Cross-session analysis | Typical outputs |
|---|---|---|---|---|---|
| Raw/native session history | What the tool shows or exports | None | One session at a time | Limited without extra tooling | Manual notes, ad-hoc fixes |
| Application-level instrumentation | Events the application emits | Code changes per app | If instrumented | Depends on the backend | Structured app telemetry |
| Telemetry-based tracing | Instrumented spans and events | Instrumentation + backend | Depends on the data model | Aggregation and queries | Traces, metrics, eval inputs |
| Proxy/provider-boundary capture | Traffic crossing the boundary | Route traffic through it | As seen at the boundary | Depends on the archive | Durable session records |
| Purpose-built session analysis | Inherited from its capture source | Varies by implementation | The primary object | Designed for it | Findings, evals, reusable artifacts |
The right approach depends on what a team is trying to do with session data. For occasional manual inspection, raw session history is enough. Teams that control their application and want structured, domain-specific events reach for application-level instrumentation. Teams that want AI activity integrated into an existing observability practice reach for telemetry-based tracing. Teams that want broad capture without instrumenting every call reach for provider-boundary capture, accepting its boundary. And teams that want to mine trajectories for improvement reach for purpose-built session analysis on top of one of those capture positions. The categories combine: a boundary-capture archive can feed a tracing backend, and a purpose-built analysis layer can read from any of them.
Whatever the approach, the mechanics converge on the same loop, and the output of analysis is broader than any one artifact type:
Captured sessions (any capture approach)
│
▼
Analysis: search, inspection, pattern review
│
▼
Findings: successful sequences, error paths, decisions
│
▼
Improvements, in whatever form fits the finding:
· debugging and code changes
· prompt or instruction changes
· evals or datasets
· tooling and infrastructure changes
· reusable procedures or skills
│
▼
Measure: did future runs actually change?This is the continuous agent improvement loop. Analysis is the step between capturing sessions and applying what they taught, and its findings can land as a bug fix, a prompt change, an eval, an infrastructure change, or a reusable procedure; a skill is one valuable output among several. An approach that stops at visibility (dashboards, traces, cost graphs) stops before the part that changes behavior. The approaches that pay off are the ones that continue past observation to changed future runs.
An example: from repeated migration failures to a reusable skill
Suppose a team’s coding agents keep stumbling on database schema migrations: a session generates a migration, applies it, and hits an ordering error because the migration assumed a schema state that no longer holds. One engineer’s session eventually resolved it: the agent read the error, inspected the current schema state before regenerating the migration, applied the corrected version, and reran the test suite to confirm.
Analysis starts with search: query the session archive for the error signature and pull up the sessions that hit it. Reading the successful session surfaces three things worth keeping: the trigger (the ordering error’s signature), the decision (inspect current schema state before writing a migration, rather than retrying blind), and the fix (regenerate against observed state, then verify with the test suite).
From there, the finding can land in more than one form: a fix to the migration tooling so the ordering error stops occurring, an eval case built from the failing sessions, or a reusable procedure extracted from the successful one. In this case the engineer packages the procedure as a skill, and a teammate reviews it, trimming run-specific literals like the database name so it generalizes. The next session that hits the same error class can activate the skill instead of rediscovering the diagnosis. Nothing in this loop required the session to be re-run or the fix to be reconstructed from memory: the recorded trajectory was the source material.
Concepts related to AI coding session analysis
- Learning from agent sessions: what to look for when you read a session record.
- Skill extraction: the step that turns an analyzed session into a reusable artifact.
- AI session capture: what each capture approach actually records.
- Agent observability: the operational discipline session analysis sits inside.
- Telemetry for agents: the data layer underneath session records.
- LLM proxy: the transport-layer position that makes proxy capture possible.
Failure modes in session analysis
- Extracting from a false success. A session can look successful without being so: the agent declared victory, but the fix was incidental. Extracting from it bakes in a pattern that does not hold. This is why review right after generation matters.
- Over-fitting to one run. Keeping run-specific literals (hostnames, file paths, one-off values) produces a skill that only fires on the exact session it came from, or quietly breaks the first time the inputs differ.
- Stopping at visibility. Dashboards and cost graphs are observation rather than improvement. An analysis practice that never produces a change to future runs is a reporting practice.
- The unread archive. Capture without an analysis habit accumulates sessions nobody reads. The archive has value only when search and review actually happen.
- Assuming completeness. Every capture approach has a boundary: the proxy path, the instrumented SDK, the terminal scroll. Treating the archive as the whole truth, rather than what crossed the capture boundary, leads to wrong conclusions about what an agent did.
When you need AI coding session analysis
Signals that a team has outgrown ad-hoc review:
- The same failure keeps appearing across different engineers’ sessions.
- “Why did the agent do that?” cannot be answered from application logs.
- The prompt library is authored from memory and has drifted from what actually works.
- New teammates’ agents re-solve problems that older sessions already solved.
- Session volume has passed what anyone can review by scrolling transcripts.
- Agent spend is rising and nobody can say which usage patterns are productive.
If none of these apply (a solo developer, a handful of sessions a week), raw transcript review is a fine starting point. The practice earns its infrastructure as volume and team size grow.
Evidence and research on AI coding session analysis
This is a young practice, and independently verified benchmark numbers on session-analysis outcomes are scarce; the only numbers this page cites are first-party findings from our own published session audits. Two grounded things can be said. First, the strongest evidence in this domain is the session record itself: unlike a claim written from memory, a statement about what an agent did (which calls it made, which error it hit, what cleared it) is checkable against the recorded trajectory. Second, the comparison above is architectural: what each capture position can observe follows from where it sits rather than from any product’s current feature list. That is why this page compares categories rather than vendors, whose capabilities change faster than a reference page should track.
How to implement AI coding session analysis
An implementation of session analysis needs three parts: a capture position, an archive that stays queryable as it grows, and a path from finding to change. One concrete implementation of purpose-built session analysis is Paper Compute’s pipeline: capture supported coding-agent sessions at the provider boundary with paperctl, preserve them as a searchable archive, inspect and analyze past trajectories in paper console, identify reusable successful behavior, generate a reviewed and versioned skill from that session evidence, then apply it to future work and measure the result. The open-source capture layer underneath, tapes, records the provider-boundary traffic, so the analysis surface and the extraction step read from one record.
Whichever implementation a team picks, the test is the same: after a month of use, can you point to a future run that went differently because of something a past session taught?