Definition
Evidence-based routing is routing that combines the semantic meaning of a request with historical evidence of what has worked for similar work, and that improves as the outcome of each decision feeds back into the next. It answers two questions together: what kind of work is this, and what has previously produced the desired outcome for work like it.
Evidence-based routing is routing that combines the semantic meaning of a request with historical evidence of what has worked for similar work, and that improves as the outcome of each decision feeds back into the next. Most routers decide from one input: either a conventional signal like price or latency, or the meaning of the request. Evidence-based routing adds a second input that the others lack, which is the record of what has actually produced the desired outcome for work of this kind.
The shift is from a question about the request to a question about the request and its history. A semantic gateway asks “what kind of work is this?” Evidence-based routing asks “what kind of work is this, and what has previously produced the outcome we want for work like it?” The first question picks a path from a fixed policy. The second picks a path from experience.
Why routing on meaning alone leaves evidence on the table
Semantic routing is a large step past routing on conventional signals, because it reads what the request is trying to accomplish. But a semantic classifier maps a request to a policy someone configured, and that policy encodes a guess about what will work. The organization usually has something better than a guess: a history of the same kind of work being run, succeeding, failing, being corrected, and costing what it cost. Routing that ignores that history re-decides from first principles every time, even when the answer is already in the record.
The organization already knows what worked for this kind of work. Evidence-based routing is the practice of routing as if it remembered.
What evidence-based routing is not
- Not routing on conventional signals. Price, latency, provider availability, model name, and rate limits are the inputs a plain gateway routes on. Evidence-based routing can respect them, but its distinguishing input is the meaning of the request plus the history of outcomes.
- Not semantic routing alone. Semantic routing supplies the classification. Evidence-based routing adds the memory of results and the feedback loop. See semantic routing vs model routing.
- Not a static learned model. A router trained once and frozen is learned routing without the evidence loop. The defining trait is that new outcomes keep updating the evidence.
- Not free of evaluation. A policy read from history is a hypothesis about new traffic. It still has to clear routing evals.
How evidence-based routing works
The mechanism is a loop rather than a single decision. A request is classified, matched against what has worked for its kind of work, run, and its outcome is folded back into the evidence for next time.
request │ ▼ what is this request trying to do? (semantic classification) │ ▼ what do we know about this kind of work? (session history / skills / context) │ ▼ what policies apply? (security, budget, latency, routing) │ ▼ which model / prompt / skill / workflow? (the routing decision) │ ▼ run it │ ▼ capture the outcome (result, cost, corrections, failures) │ ▼ use that evidence to improve the next decision
The loop is what separates evidence-based routing from a one-shot classifier. Each run is both a decision and a new data point. The routing policy is never finished; it is the current best reading of an accumulating record.
The signals evidence-based routing can draw on
The evidence is only as rich as the record behind it. A full session history can carry far more than a request and a response.
| Task and intent | What the work was, the basis for matching a new request to prior work of the same kind. |
|---|---|
| Models and prompts used | Which model and prompt handled it, so successful pairings can be preferred. |
| Tool calls and reasoning | The path taken through tools and context, in addition to the final answer. |
| Corrections and failures | Where the work went wrong and had to be redone, the negative evidence a benchmark lacks. |
| Human pushback | Where a person rejected or rewrote the output, a strong signal of a bad outcome. |
| Cost and turns | What the outcome cost and how many turns it took, so cheaper paths to the same result surface. |
| Outcomes and skills | Whether the work succeeded, and any skill extracted from work that did, as reusable evidence. |
Corrections, failures, and human pushback are the signals that make this different from a public benchmark. A benchmark records what a model produced; an organization’s own history records what happened next, including whether a human had to fix it. That negative evidence is often the most useful input a router can have.
Cost optimization as an evidence problem
The strongest case for evidence-based routing is cost, framed carefully. The goal is to find the cheapest model that produces the desired outcome for this kind of work, which is a harder question than finding the cheapest model overall.
The goal is the cheapest model that still produces the outcome you want for this kind of work.
Answering it needs evidence, because whether a cheaper model produces the desired outcome depends on the kind of work, and sometimes the cheaper model is genuinely better for a specific kind. A history that records outcomes and cost together lets a router find, per kind of work, the least expensive path that has actually worked, rather than assuming cost and quality trade off the same way everywhere. See cost-aware model routing for the spend side of this in isolation.
Example: routing a class of work the org has seen before
A team’s agents repeatedly do a specific kind of work: turning a support ticket into a drafted reply. Over months, the session history accumulates hundreds of these, with outcomes attached: which drafts shipped unedited, which a human rewrote, which model wrote each, and what each cost.
A semantic router would classify a new ticket as “draft a support reply” and send it to whatever model the policy names. An evidence-based router classifies it the same way, then consults the history for that class: the mid-tier model produced ship-ready drafts on this kind of ticket as often as the frontier model, at a fraction of the cost, while a specific cheap model’s drafts were rewritten by a human most of the time. It routes to the mid-tier model, captures whether this draft ships, and adds that to the record. The decision was a reading of what had worked rather than a guess about difficulty.
Concepts related to evidence-based routing
- Semantic gateway: the decision layer that supplies the classification half.
- Using agent session data for model routing: the captured history that supplies the evidence half.
- Model routing: the parent concept.
- Cost-aware model routing: the spend framing this sharpens into an outcome question.
- Model routing evals: how a policy read from history is validated on new traffic.
Failure modes of evidence-based routing
- No corpus, or no outcomes in it. Without a history there is nothing to consult, and a history of what was tried without what worked cannot ground a decision. The record has to carry outcomes.
- Cold start. For a kind of work the organization has never done, there is no evidence, and the router has to fall back to semantic or conventional routing until history accumulates.
- Feedback-loop bias. If the router only ever tries the paths history already favors, it never gathers evidence about the alternatives, and the policy ossifies. It needs occasional exploration to keep the evidence honest.
- Stale evidence. History from before a model or workflow changed can point at a path that no longer works, the routing version of drift.
When you need evidence-based routing
Reach for it when you have both a repeating body of work and a record of how it has gone:
- The same kinds of work recur often enough that history is informative.
- Your session records carry outcomes alongside requests and responses.
- Cost or quality matters enough that “what has worked for this kind of work” beats a static policy.
- You can close the loop, feeding each decision’s outcome back into the evidence.
If your work is one-off, or you have no record of outcomes, evidence-based routing has nothing to stand on, and semantic or cost-aware routing is the right starting point until a history exists.
Evidence and research
Evidence-based routing sits at the join of two documented lines of work. The learned-routing line, exemplified by RouteLLM (LMSYS and Anyscale, ICLR 2025, arXiv 2406.18665), shows that a routing policy derived from outcome data can hold quality while cutting cost. The semantic-routing line, exemplified by the vLLM Semantic Router, shows routing driven by the meaning of a request. Evidence-based routing is the combination: the semantic classifier decides the kind of work, and an outcome record decides what has worked for it, with the two kept current by a feedback loop.
How Paper Compute relates to evidence-based routing
One implementation depends on having the record first. Paper Compute captures the model traffic that passes through its gateway into a durable tapes store: the prompts, the responses, the models used, and the tool-call messages that travel inside that traffic, with cost and turn counts derived from it. Corrections and outcomes appear in that record only when they pass through the traffic, such as a rewritten draft sent back as the next prompt. That record is the corpus evidence-based routing would draw on. Turning it into routing decisions is a workflow on top of the capture rather than a property of it, and it still needs classification, a policy, and evaluation.
paper is currently in development. Sign up for the waitlist to get early access.
Last verified: September 10, 2026.