TypeSafe’s Jev returns decisions and probabilities. I wanted to test it as a quick judge for labeling agent sessions. I wrote an experimental labeler and ran the same workload through Jev, GPT-4o mini, Claude Haiku 4.5, and a local Qwen model: 1,781 engineer turns from 138 sessions, 30 days of my team’s work. Jev answered in a 188 ms median and recovered 100% of the human-accepted question-label examples.
Paper Compute’s labels feature already supports labels down to individual spans, the recorded operations inside a trace. High-fidelity traces preserve the instructions, tool calls, and results behind those operations. Together, they give me a way to find useful work and return to its context. My Jev labeler is a separate experiment in proposing those labels.
Four questions, four models
I captured reviews happening inside my team’s agent sessions: a teammate questioning the agent’s work, pushing back, or giving it direction. These are interactions with the agent, not GitHub pull-request reviews. I started with a fixed set of 35 reviewed turns, then widened the corpus to every judgeable engineer turn from the last 30 days: 1,829 turns across 138 sessions. I dropped 48 turns whose request bodies were corrupted in transit, for every model, leaving 1,781. Each model received the same turn context and four questions: did the engineer push back, ask about the agent’s work, state a lasting fact or preference, or supply knowledge that would have changed the previous action?
Those questions produce candidate pushback, question, observation, and missing_knowledge labels. With Jev, I used Noul questions, which each return a probability of true. The Jev explainer covers how that works.
Jev returned a decision in 188 ms
That was the median across 1,781 turns, including parsing and validation. GPT-4o mini took 1,023 ms and Claude Haiku 4.5 took 1,050 ms, about 5.4 and 5.6 times longer. Qwen3 8B, running locally, took 2,311 ms. The pilot on 35 turns had put Jev at 176 ms; the larger sample moved it by 12 ms.
How long did a decision take?
Complete responses, including parsing and validation, one call per turn, no retries. Jev, Haiku, and Qwen3 8B: 1,781/1,781. GPT-4o mini: 1,779/1,781; two client timeouts. Cloud runs and the local run happened in parallel on 2026-09-23; Qwen3 8B shared the workstation with them, so its times are an upper bound. Initial requests excluded.
The p95 shows the slower end of each run. GPT-4o mini reached 2,277 ms and Qwen3 8B 3,580 ms, compared with Jev’s 273 ms. Jev, Haiku, and Qwen3 8B completed all 1,781 requests. GPT-4o mini completed 1,779; two timed out on my side and were not retried.
Jev found 100% of accepted question labels
The 35 reviewed turns sit inside the 30-day corpus, so the agreement check is the same as before with more surrounding data. Of the 25 reviewed question-label proposals, 19 had been accepted and six rejected. Jev recovered all 19 accepted examples: 100% recall. It also flagged the six rejected examples, giving it 76% precision.
What did each model find?
| Model | Accepted recovered / 19 | Rejected also flagged / 6 | Flags a reaction |
|---|---|---|---|
| Jev 1.13.0 | 19 / 19 | 6 / 6 | 53% |
| GPT-4o mini | 16 / 19 | 5 / 6 | 82% |
| Claude Haiku 4.5 | 18 / 19 | 5 / 6 | 67% |
| Qwen3 8B | 9 / 19 | 3 / 6 | 12% |
Question-label subset: 19 accepted and 6 rejected proposals, the same reviewed turns as the pilot, now inside the 30-day corpus. These examples helped refine the questions; this is development-set agreement, not held-out accuracy. The flag rate counts every turn where the model's combined pushback, question, and observation probability reached 0.5.
Claude Haiku 4.5 recovered 18 of the 19 accepted examples and flagged five rejected ones: 95% recall and 78% precision. GPT-4o mini recovered 16 and flagged five: 84% recall and 76% precision. Qwen3 8B found nine and flagged three rejected examples: 47% recall and 75% precision.
The unreviewed turns cannot score accuracy, but they show how often each judge fires. Jev flagged a reaction on 53% of the 1,781 turns, Haiku on 67%, GPT-4o mini on 82%, and Qwen3 8B on 12%. Roughly two thirds of the reviewed turns were real reactions, so GPT-4o mini’s extra flags are landing on turns where nothing happened, and Qwen3 8B is missing most of the ones that did. I also tried Qwen3 0.6B. It answered in 446 ms but flagged every turn and recovered three of the 19 accepted examples, so it is not in the figures.
I had already used the reviewed turns to refine the questions, so this is a development set. The results measure agreement on those reviewed examples. Unlabeled kinds remain unknown, and I still need a fresh evaluation set to measure how well the labels generalize.
How I ran the comparison
Jev was requested as jev-latest and resolved to jev-1.13.0. I ran qwen3:8b and qwen3:0.6b locally, both Q4_K_M, through Ollama 0.17.4 on an Apple M4 Max with 36 GiB of memory. GPT-4o mini ran through OpenAI’s Responses API as gpt-4o-mini, resolving to gpt-4o-mini-2024-07-18. Claude Haiku 4.5 ran through Anthropic’s Messages API as claude-haiku-4-5, resolving to claude-haiku-4-5-20251001. The corpus and question hashes matched across runs.
The corpus is every engineer turn with a previous agent turn to react to, from sessions active in the 30 days before 2026-09-23. Each request contained a redacted view capped at 1,500 characters of the engineer’s message, 800 characters of the agent’s previous response, and 12 descriptions of actions. I included a short session title and harness identifier, excluding tool outputs, file contents, first turns, and synthetic recap prompts. Forty-eight turns never reached Jev because their request bodies were blocked in transit; I treated them as corrupted and removed them from every model’s set. The benchmark tested label decisions on these excerpts. It did not test selecting spans or attaching labels automatically.
Each model received one initial request, then one serial pass over the 1,829 turns in the same shuffled order. I excluded the initial request from latency measurements, disabled the application’s verdict cache, and did not retry failures. Provider-side and local prompt caching could still occur; GPT-4o mini reported 78,336 cached input tokens.
The local requests used JSON-schema output, thinking disabled, temperature zero, an 8,192-token context, and a 128-token output cap. GPT-4o mini and Haiku received the same system and user text as the local models, with JSON-schema output, temperature zero, and the same 128-token output cap. I set store: false on OpenAI; provider retention policies still apply.
The times include response parsing and validation. The hosted models’ times also include the network and TLS path; the HTTP client did not use a persistent connection pool. The three hosted models ran as one serial process and the local models as another, at the same time, on a workstation that was also running other work, so the local times are an upper bound. Network conditions, provider load, and local activity could affect the results.
I refined the questions before this comparison. My first version used five broad intents. I replaced those with narrower questions, then expanded the question definition to include checks about the agent’s work: whether a feature was finished, what a change affected, or whether something actually ran.
My code chooses the highest-scoring reaction kind and applies a combined reaction threshold. This is a selection heuristic; the combined score and the generative models’ probability estimates have not been calibrated. The missing_knowledge judgment also needs further review to establish whether missing information caused a mistake.
What those judgments cost
Speed is the main reason I am interested in Jev for this task. The API cost also leaves room to run these judgments repeatedly. The figures below use TypeSafe’s published list price, GPT-4o mini’s standard rates, and Claude Haiku 4.5’s list price, checked for the runs.
- per million input tokens
- $0.042
- Output tokens are free
- known-usage estimate
- $0.186
- 4,426,088 input tokens, including warmup
- measured input tokens
- 4,423,583
- 1,781 successful measured responses
Free output. Extrapolated from successful measured usage, including repeated question definitions. Not an invoice or a measured volume run. Capture, review, hardware, and energy excluded; local inference is not free.
Including the initial request, Jev reported 4,426,088 input tokens for an estimated $0.186. GPT-4o mini reported 3,876,724 input tokens, 79,872 of them cached, and 49,595 output tokens for $0.605. Claude Haiku 4.5 reported 4,502,038 input tokens and 65,560 output tokens for $4.83. These estimates use each provider’s reported usage.
At the observed response sizes, 1,000 similar successful responses would cost about $0.104 with Jev, $0.330 with GPT-4o mini, or $2.63 with Haiku. Jev is roughly three times cheaper than GPT-4o mini and 25 times cheaper than Haiku on the same inputs. Local hardware, electricity, and human review costs are outside this comparison.
Finding human direction in a trace
Two labels I care about are pushback and apology. Pushback captures the human in the loop adding value: correcting the agent and teaching it how the work should be done. The apology that follows marks the agent acknowledging that direction. These exchanges are useful data for generating skills from sessions, so the correction can guide the agent the next time it does similar work.
These illustrative sessions can be filtered by either label or both.
Session | Status | Name | Labels | User | Turns | Cost | Active |
|---|---|---|---|---|---|---|---|
| a3f8c21d | Running | keep validation in the request handler ~/code/papercomputeco/console | brian(You) | 42 | $18.40 | 12m ago | |
| d91b04ee | Completed | rerun tests after the parser correction ~/code/papercomputeco/tapes | maya | 11 | $2.10 | 1h ago | |
| 7c25e9ba | Completed | acknowledge the missed UTC convention ~/code/papercomputeco/tapes | brian(You) | 67 | $31.75 | 3h ago | |
| f04a77c3 | Completed | scaffold billing exporter cron ~/code/papercomputeco/infra | — | devon | 24 | $6.88 | 5h ago |
| c18e5a09 | Failed | restore the check removed by the agent ~/code/papercomputeco/tapes | maya | 19 | $4.02 | yesterday |
Labels reach all the way down to a span. That lets me mark the specific operation the engineer corrected and inspect it alongside the surrounding exchange. This is a practical use for the captured work I discussed in Dreams, Reflections, and Inceptions.
The examples below show the labeling task I gave Jev. They are invented examples, not recorded model outputs. Apologies are useful in the existing labels workflow; this benchmark’s four questions focused on the engineer’s message.
Where Jev fits
Jev delivered 100% recall on the 19 accepted question-label examples, with a 188 ms median response across 1,781 turns and 138 sessions. That combination of coverage and speed is why I want Jev in the labeling workflow. Human review still matters: it also flagged six rejected examples, and it flags roughly half of all turns, so the queue it produces is a review queue, not a verdict.
I already use labeled spans in a separate experiment to train a small code-review model to flag a change or let it pass. That is the connection: labeled spans provide examples of human judgment, and this experiment compares ways to find them quickly.
The data I used is private to my team. If you want to run a similar test on your own sessions, autolabeling is being tested and will ship soon. Reach out if you want early access.
Related reading
Start with paper
Turn every session into knowledge at team scale.