Definition
An AI gateway is a single point on the network that model requests pass through on their way to a provider. Requests routed through it can be authenticated, routed, checked against policy, and recorded; requests sent directly to the provider skip all of that.
An agent is asked to draft a reply to a customer’s double-charge complaint. Two seconds later, a good reply appears. That exchange is our specimen for this whole series, and the question this first page asks about it sounds almost too simple: on its way to the model, where did the request actually go?
The answer matters more than it sounds like it should, because everything an organization can know, control, or afford about its AI usage is decided by the route.
Why organizations reach for an AI gateway: key sprawl
Start with what happens without one. Each AI tool in an organization connects itself to each model provider it uses: its own API key, its own configuration, its own idea of which model to call. With two tools this is fine. Slide the figure below toward eight and watch what the wiring turns into. Every line you see is a credential someone has to store and rotate, a policy that has to be re-implemented, and a stream of spending nobody sees in one place.
Every tool wires itself to every provider, and every wire is a credential to manage, a policy to duplicate, a place spend hides. Slide to 8 tools, then add the gateway.
That collapse from a tangle to a hub is the original reason this category of infrastructure exists; the rest of the series shows what else the one doorway makes possible. The mailroom is the useful analogy: a building full of people could each walk their own letters to the post office, but the building installs a mailroom instead, because one doorway is the only arrangement where logging, checking, and sending can happen consistently. A gateway is the mailroom for model requests.
The two routes a model request can take: direct or through the gateway
Now put our specimen request on the wire and watch it travel. Routed through the gateway, it crosses five stops before reaching the provider, and the moment the model actually runs, called inference, is the only step the gateway does not do itself. Hover each stop to read its job; click one to see how it works and what breaks without it. Then flip the route to direct and watch the two counters at the top.
Every dot crosses five stops, and the shared record grows with the traffic. Hover a stop to read its job; click it to see how it works and what breaks without it.
Two things are worth noticing before moving on. First, on this plain forwarding route the reply is the same on both paths, at nearly the same speed: the gateway adds a hop, and none of these five stops rewrites the answer. (The next page adds stages that can change the prompt the model receives.) Second, the counters split the moment you go direct. Requests keep being served while the shared record stops growing. Fragments of the direct exchange may survive somewhere (the provider keeps usage records in its own console; the tool may write a log on its own machine), but each fragment lives in a different account under someone else’s retention rules. Nothing lands in a place the organization can query as one record.
What the gateway’s shared record makes answerable
A record nobody reads is just storage. The reason the capture stop matters is what it makes answerable afterward. Below is the record the gateway wrote for one hour of the specimen’s day: five requests, each with its time, tool, model, tokens, and cost. Ask it the questions on offer. Then flip the toggle so the same hour of traffic went direct, and ask again.
| time | tool | model | tokens | cost |
|---|---|---|---|---|
| 14:02 | chat | mid | 1.2k | $0.004 |
| 14:05 | coding agent | frontier | 18k | $0.062 |
| 14:06 | chat | small | 0.3k | $0.001 |
| 14:11 | coding agent | frontier | 22k | $0.081 |
| 14:19 | chat | in-house ·sensitive | 0.9k | $0.003 |
This is the record the gateway wrote (five requests from the specimen's hour, illustrative). Pick a question, then flip the toggle and ask it again.
That flip is the punchline of this page. With the rows written, “what did this hour cost?” is a query that takes a second. Without them it is an investigation that stitches together provider invoices and scattered logs, and some questions (did sensitive data stay inside?) simply have no answer at all. The gateway does not make the organization smarter; it makes the questions cheap.
The next page opens the gateway itself: the request does not just pass through a box, it falls through a sequence of decisions, and which model it lands on depends on what it is asking for. For the reference treatment of everything on this page, see AI gateway; for the interception mechanics one layer down, see LLM proxy.