Definition
Inside a gateway, a request passes through a sequence of stages before inference: it is captured, its intent is classified, policy and budget are checked, a route is chosen, and a specific model and prompt are selected. The stages compose like stations on a line, and new ones can be added without moving the endpoint.
The last page followed our specimen request to the gateway’s front door and treated the inside as five quick stops. This page opens the door properly. Inside, a gateway is less like a checkpoint and more like a waterfall: the request enters at the top, falls through a sequence of stages, and lands on a model. Which model, with which prompt, depends on what the request is asking for.
The stages inside an AI gateway pipeline, one at a time
Before watching anything move, it is worth knowing what each stage actually is, because none of them is magic. Capture writes the request down. Classify reads it. Policy checks the rules. Route picks a path, select attaches the exact model and prompt, and fuse (rare, optional) can blend two models’ drafts into one reply. Hover each stage below for its mechanics; click it to see what breaks when it is missing, which is the fastest way to understand why it exists.
Read what kind of work this request is asking for.
how: A small, fast classifier labels the request in milliseconds: quick fact, draft, hard reasoning, sensitive data.
click the stage to see what breaks without it
Notice the shape of those “without it” answers. Remove capture and you lose the record from the last page. Remove classify and the gateway goes static. Every stage is the answer to a specific failure, which is exactly how real gateways accumulate them.
How intent classification routes requests to different models
The stage doing the most interesting work is classify. A request’s intent is what it is trying to get done: look up a fact, draft a message, reason through a hard problem, handle something private. A person triaging a shared inbox does this instinctively; the classify stage is that triage, performed by a small, fast classifier in milliseconds, before any expensive model runs. Routing on that label has a name, semantic routing, and the figure below is what it looks like: four intents raining through the same stages and branching into four different model lanes. Spotlight a color and follow it down.
Four intents rain through the same stages and land in four different lanes. Spotlight a color to read its routing decision. One to try: can you make the sensitive color land anywhere except in-house? (You cannot. That is policy outranking price.)
Try the challenge in the caption: the sensitive color cannot be made to land anywhere except the in-house lane. That is policy outranking routing, and it is the correct hierarchy. Price and capability get a vote on where work goes; the rules about what may leave the building get a veto.
How a gateway pipeline grows one capability at a time
Here is the part that explains why gateways look so different from company to company. Nobody installs all of this at once. A real gateway starts as a bare forwarding pipe, and stages get added one at a time, each because someone hit the failure it prevents. The figure below starts you with the bare pipe. Add each capability and read what it unlocks; the scene changes with every addition.
- Right now every request, whatever its color, takes the one path, and nothing is written down. Add a capability and read what it unlocks.
Start with the bare pipe, then build. The order you add things here is a common one; real gateways grow in the order their failures arrive.
Two things fall out of building it by hand. First, the order is a common one: capture first (you cannot improve what you cannot see), routing second (the bills from the next page force it), policy third (the first near-miss with sensitive data forces that). Real gateways grow in whatever order their failures arrive, and some start with policy. Second, and this is the durable idea: the endpoint never moved. Tools kept pointing at the same address while the pipe underneath became a gateway. That is why the pipeline can keep growing (caching, insights, fusion, stages nobody has named yet) without any tool changing a line of configuration.
The next page zooms into the routing decision and puts real prices on it: when the cheap lane is right, and when it quietly costs the most. For the reference treatment, see semantic gateway and semantic routing vs model routing.