Definition
Routing in an AI gateway is the per-request decision of where traffic goes. Every routing scheme answers two questions: which signal drives the decision (static configuration, prompt semantics, a learned quality score, availability, latency, budget, or policy) and which destination the decision resolves to (a model, provider, endpoint, tool, cache, workflow, or rejection). Semantic routing names schemes whose signal reads the request; model routing names schemes whose destination is a model choice. The two overlap wherever a prompt-conditioned signal picks a model.
A request arrives at a gateway, and something decides where it goes: which model, which provider, sometimes a cache or a tool instead. Semantic routing and model routing are the two names people reach for when they talk about that decision, and they get used interchangeably. The tidy version says semantic routing is a strategy and model routing is a target, and it turns out to be too neat: real routers mix signals, and the most cited “non-semantic” routers read the prompt. The model that holds up has two axes: what signal drives the decision, and what destination the decision resolves to.
The two axes of AI gateway routing: decision signal and destination
Every routing scheme answers two questions: what does the decision read, and where can it send the request?
| Signal | What it reads | Example |
|---|---|---|
| Static configuration | Model name, caller identity, fixed rules | Model-name virtualization; per-team backends |
| Prompt semantics | Classified intent, topic, complexity, or sensitivity | An intent classifier picks the path (vLLM Semantic Router) |
| Learned quality score | A trained predictor of whether the cheaper path suffices | RouteLLM win-rate routers |
| Availability | Provider health and error rates | Provider fallback lists |
| Latency | Queue depth, time-to-first-token targets | Latency-aware endpoint picking |
| Budget | Remaining quota and spend targets | Cost-cap downgrades near end of budget period |
| Policy | Data class and compliance rules | Sensitive prompts routed to an internal model |
| Destination | What resolving there means |
|---|---|
| Model | A stronger or cheaper model serves the request |
| Provider | The same model class through a different vendor or region |
| Endpoint | A specific replica, adapter, or serving configuration |
| Tool | The request is answered by a tool or function rather than generation |
| Cache | A stored response for a near-identical request is returned |
| Workflow | The request enters a multi-step pipeline: reasoning mode, retrieval, review |
| Rejection | The request is refused or routed to human review under policy |
The two common terms each name one axis. Model routing names the destination column: the decision resolves to a model choice. Semantic routing names a signal family: the decision reads the request. The overlap region (a prompt-conditioned signal picking a model) is where most production routing lives, which is why the two terms blur.
Ask two questions of any router: what does the decision read, and what can it resolve to. The marketing label answers neither.
Why the semantic vs model routing split is not categorical
The clean split says a cost-threshold router is non-semantic because it optimizes a knob rather than reading meaning. That case is real. It stops being clean one step later, with learned routers.
RouteLLM, the open-source framework from LMSYS and Anyscale, is the standard citation for model routing. Its routers read the prompt: each one takes the user prompt and predicts how likely the strong model is to win on that specific prompt. The trained variants include a matrix factorization model, a similarity-weighted Elo ranking over embedded queries, a BERT classifier, and an LLM-based classifier. A learned score computed from prompt features is a semantic signal in every sense that matters, even though it never produces a readable intent label.
So the distinction that holds up inside the signal axis is between interpretable intent classes (a classifier says “trivial lookup” or “legal reasoning,” and policy maps each class to a path) and learned scores (a model produces a number, and a threshold maps it to a path). Both read the prompt. They differ in how easy they are to inspect and debug, and in how policy attaches to them, rather than in whether they are “semantic.”
Examples across the routing space
Three designs cut spend on the same mixed workload, and they sit at different points in the space:
- Threshold on a static knob. A fixed fraction of traffic goes to the small model. Signal: budget. Destination: model. It reads nothing, so it cannot protect hard requests from downgrade beyond tuning the fraction.
- Learned score. A RouteLLM-style router predicts per prompt whether the weak model will suffice. Signal: learned quality score (prompt-conditioned). Destination: model. Reported results reach cost reductions of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K while retaining roughly 95% of strong-model performance; the authors’ benchmark figures rather than a guarantee.
- Intent classes. A small classifier labels each request, and policy maps labels to paths: trivial lookups to a cheap model, reasoning-heavy tasks into a reasoning workflow, sensitive content to an internal model, near-duplicates to a cache. Signal: prompt semantics. Destinations: model, workflow, cache, and policy-driven rejection. The vLLM Semantic Router is the open reference for this shape, and What is a semantic gateway covers deploying it at the gateway.
How to measure routing quality
Routing writing is long on architecture and short on evaluation. A router is a decision system, and it needs the same discipline as any other one:
- False escalation rate. Easy requests sent to the expensive path. Pure cost leak, invisible in quality metrics, visible in spend per resolved request.
- False downgrade rate. Hard requests sent to the weak path. This is the dangerous direction because it degrades quality silently; detecting it requires a quality signal (evals, user feedback, escalation-on-retry patterns), and a safe design fails uncertain cases toward the stronger path.
- The cost-quality frontier. A router is a curve, with each threshold setting trading cost against quality. Evaluate the curve against a random-mixing baseline between the same endpoints; a router that does not beat random mixing is overhead. RouteLLM’s evaluation reports exactly this shape: percent of strong-model performance achieved at a given percent of strong-model calls.
- Shadow evaluation. Run a candidate router on live traffic without letting it act, log its decisions alongside actual outcomes, and compare. This is the routing version of shadow mode for policy, and it requires captured traffic to exist first.
- Per-intent performance. Aggregate numbers hide category regressions. A router can improve the average while collapsing on one class (code generation, one language, one team’s workload). Slice the frontier per intent class before trusting it.
All five measurements need the same thing first: a captured record of requests, decisions, and outcomes. Routing evaluation is a downstream consumer of the gateway’s capture archive.
Where routing sits: the decision layer and the control plane
The routing layer decides the path. The gateway’s control plane governs and records the traffic on whichever path was chosen. The two work together in one request path, even though they are sometimes marketed as rival products.
caller │ ▼ ┌───────────────────────── AI GATEWAY ─────────────────────────┐ │ │ │ authenticate ──► ROUTING DECISION ──► policy & egress │ │ (control plane) signal → destination (control plane) │ │ │ │ forward upstream │ │ │ │ capture + meter the exchange AND the routing decision │ └───────────────────────────────────────────────────────────────┘ │ ▼ provider(s) / cache / tool / workflow
The order matters. Identity comes first because routing policy may depend on who is calling. The routing decision picks the path. Policy, egress, and capture wrap the forwarded request. The routing decision itself lands in the captured record, and that is what makes shadow evaluation and per-intent slicing possible at all. Deploy only half of this diagram and you get one of two familiar failures: a smart router that remembers nothing, or a well-governed pipe that overpays on every easy request.
Concepts related to semantic and model routing
- What is a semantic gateway: deploying content-aware routing at the gateway, including its evaluation and safety obligations.
- AI gateway: the pillar; routing is one of its five responsibilities.
- LLM proxy: the data-plane mechanism the routing decision executes through.
- Enterprise AI gateway: operating the governed path routing runs inside.
Failure modes in routing design and evaluation
- Buying the label instead of the axes. A router marketed as semantic may read nothing but a cost threshold. Ask what the decision reads and what it can resolve to.
- Expecting a two-destination router to make multi-destination decisions. A strong-versus-weak router cannot turn a request into a cache hit or a tool call. Those need a signal with more classes and a policy with more destinations.
- Evaluating a point instead of the frontier. One cost number at one threshold says nothing about the router. Plot the curve, compare against random mixing.
- Ignoring the silent failure direction. False downgrades do not appear in cost dashboards. Without a quality signal, a router can look like pure savings while degrading the hardest requests.
- Routing without capture. If decisions are not recorded alongside outcomes, none of the five quality measurements above are possible, and the router is unfalsifiable.
When to reach for each routing signal
- Static configuration is enough while one team hand-tunes one workload. It is also the fallback every other signal degrades to.
- A learned score is the lightest lever when the target is cost on a two-model setup and you can tolerate a scalar you cannot easily interrogate.
- Intent classes earn their complexity when destinations multiply beyond models, when policy has to attach to content (sensitivity, compliance), or when you need to debug why a request went where it did.
- Availability, latency, and budget signals compose with all of the above; production routers are pipelines, with policy checks first, semantic or learned selection second, and health-based fallback last.