Definition
An AI gateway is a service that sits between a company's AI tools and the model providers they call. Tools send their requests to the gateway instead of directly to the provider, and the gateway forwards them. Because the traffic passes through it, the gateway is where a platform team chooses which model serves each request, enforces company rules, records what was sent and returned, and tracks cost. It can only do this for traffic that is actually routed through it.
Normally, an AI tool sends its requests straight to a model provider. Your coding agent calls the Anthropic API. A chat app calls OpenAI. Each tool holds its own API key, and nothing sits in between.
An AI gateway changes one thing about that picture: tools send their requests to a single endpoint the company controls, and that endpoint forwards them to the provider.
without a gateway: AI tool ─────────────────► model provider with a gateway: AI tool ──► AI GATEWAY ──► model provider
That one change is the whole idea. Requests now pass through a place the company controls, so the company can make decisions there: which model handles each request, what is allowed to be sent, what everything costs, and what actually happened during a session. The rest of this page unpacks that box.
Why teams put a gateway between tools and providers
Almost nobody starts with a gateway. AI use starts as a few engineers with personal API keys, and it grows into dozens of tools calling several providers. Then the questions arrive:
- Finance asks what the company spent on AI last month, by team. The only answer is a pile of provider invoices and a card statement.
- Security asks which tools sent company data to which providers. Nothing was watching that path.
- An application team wants to switch providers, or survive a provider outage, without editing every service that calls a model.
No single AI tool can answer these questions, because no single tool sees the whole picture. A gateway can, because all of the traffic passes through it. When two or three of these questions arrive in the same quarter, that is the signal to adopt one. The trigger is the questions rather than any particular team size.
The gateway exists because every hard question about AI usage has the same answer: it happens at the place the traffic passes through.
Example: a coding agent behind an AI gateway
Here is the change from the developer’s side. A team runs a coding agent that talks to a frontier model. The platform team points the agent at the gateway by changing one setting, the base URL. The agent’s code does not change.
Now follow one afternoon of requests. The agent sends a request to the gateway. The gateway checks who is calling and confirms the request is allowed. It sees the request is routine work and sends it to a cheaper model, saving the frontier model for the hard turns. When the primary provider has an outage an hour later, the gateway retries against a healthy one and the session continues. The company keeps one record of every request, response, and cost. And the developer’s personal API key is gone: the gateway holds the provider keys now, so offboarding an engineer no longer means hunting down credentials.
That short story is the whole job description. Naming what it demonstrated:
| Responsibility | What it answers |
|---|---|
| Route | "Which provider and model should serve this request, and what happens if that provider is down?" |
| Govern | "Is this caller allowed to make this request, and did anything leave that should not have?" |
| Observe | "What was sent, what came back, and what did the agent do during that session?" |
| Control cost | "What did this team, model, and project spend, and is it within budget?" |
| Secure | "Where do provider credentials live, and who can reach which backends?" |
What an AI gateway does on each request
Follow one request through the box. Each stage answers one operational question, and each maps to a concept you can read on its own.
Pick a stage above to see the operational question it answers and the concept it maps to.
The order is logical rather than rigid. The gateway first checks who is calling, because everything else depends on that. Then it picks a provider and model. Then it applies company rules: some content is redacted, some requests are blocked, some are over budget. Recording happens on both sides of the provider call: the gateway records the request before sending it out, then records the response and counts tokens as it comes back.
Real gateways choose which stages to run, and skipping one is a legitimate choice. It just means that responsibility lives somewhere else, and it is better to decide that on purpose than to discover it during an incident.
caller (tool / agent / service)
│ request in the provider's API format
▼
┌─────────────────────────────────────────┐
│ AI GATEWAY │
│ │
│ 1 Authenticate & authorize the caller │
│ 2 Route to a provider / model │
│ 3 Apply policy & redaction rules │
│ 4 Enforce budget & rate limits │
│ 5 Record the request, forward it │
│ 6 Record the response, meter tokens │
└─────────────────────────────────────────┘
│ ▲
▼ │
provider(s): Anthropic, OpenAI, Bedrock, Vertex, self-hostedWhat an AI gateway can see, and what it cannot
A gateway sees the traffic that flows through it: prompts, responses, and the tool-call messages that travel inside them. That is the raw material for every record, report, and replay the gateway produces.
Knowing what the gateway cannot see matters just as much:
- Work that happens on the machine. When an agent runs a shell command or edits a file, only the message describing that action crosses the provider connection. The actual execution and its side effects never pass through the gateway.
- Traffic that skips the gateway. A tool holding its own provider key can call the provider directly. The gateway never knows that request existed.
- Browser AI. ChatGPT or Claude open in a browser tab talks to the vendor’s own servers. None of it touches the gateway.
This matters most when reading vendor claims. “Captures every tool call” really means “captures every tool-call message that passes through the gateway.” The gaps get closed with other controls (device management, network rules, runtime isolation) rather than assumed away.
Coverage: making model traffic actually pass through the gateway
A gateway only governs the traffic that reaches it. “Every request passes through the gateway” is a goal you engineer toward rather than a fact that comes with the install. The classic counterexample is a script on someone’s laptop with an old provider key in its environment, calling the provider directly for months. Four mechanisms close that door:
- Hold the keys at the gateway. If tools never carry their own provider keys, the gateway is the only way to get a response.
- Point managed tools at the gateway. Base-URL settings, config management, or device profiles make the gateway the default path.
- Watch the network. Block or flag direct connections to provider endpoints from inside the company network, so traffic that skips the gateway becomes visible instead of silent.
- Be honest about browser AI. Chat apps on vendor domains never touch the gateway. They need browser-level or policy-level controls of their own, and pretending the gateway covers them is how audits get surprised.
How AI gateway routing works across providers and models
Routing is the gateway deciding where each request goes. It happens at two levels.
Static routing follows rules written down ahead of time: this model name goes to this provider, this team’s traffic uses this backend. Two staples are worth knowing by name. Model-name virtualization lets tools ask for a generic name (say, default-coding-model) that the gateway resolves to a real provider model; change the mapping and every tool switches models with no code change. Provider fallback is a ranked list of providers, where traffic moves to the next entry when the current one returns errors. The open-source Envoy AI Gateway documents both.
Dynamic routing decides per request, using something about the request itself: its cost target, its difficulty, or its content. Reading the request to choose its path is the subject of Semantic and model routing, and What is a semantic gateway covers running that capability at the gateway, including the evaluation and safety work it creates.
AI gateway reference architecture: control plane, data plane, trust boundaries
Inside a production gateway there are really two systems, and they have standard names.
The part that sits on the request path is the data plane. It is the network process that receives each request, records it, applies the rules, and forwards it to the provider. That process is called an LLM proxy, and every gateway has one at its core. The proxy page covers it in depth.
Everything that manages the proxy is the control plane: routing tables, policy definitions, budgets, the stored provider credentials, and the queries you run over captured traffic. When an admin adds a budget rule in a dashboard, that is the control plane. When a request gets checked against that budget mid-flight, that is the data plane.
Keeping the two separate is what lets a team change a policy without touching the request path, and scale the request path without rebuilding governance.
One more line on the diagram matters: the trust boundary, the edge between what the company controls and what it does not. The gateway sits exactly on that edge, which is why policy and redaction run there.
CONTROL PLANE
routing config · policy · budgets · credentials · archive queries
│ configures ▲ records
▼ │
┌─────────────────────────────────┐
callers ───► │ DATA PLANE (LLM proxy) │ ───► model providers
│ authenticate · route · apply │
│ policy · record · meter │
└─────────────────────────────────┘
(inside the company) (outside the company)
the data plane sits on the trust boundaryThe blind spots from earlier belong in this picture too, drawn separately so the flow stays readable:
traffic that skips the gateway entirely: · tools calling providers directly with their own keys · browser-hosted AI on vendor domains · personal accounts on unmanaged devices invisible even for routed traffic: · local execution between turns: the command actually run, the file actually written, and its side effects · anything that never crosses the provider connection
One taxonomy: AI gateway, LLM proxy, semantic routing, inference gateway
By this point you have met all the pieces. The vocabulary, quickly:
- AI gateway: the broad system this page describes. LLM gateway and model gateway are rough synonyms.
- LLM proxy: the piece that sits directly in the request path. The data plane above.
- Semantic routing: one capability a gateway may use to decide where a request goes, by reading the request itself.
- Inference gateway: an ambiguous term that different communities use differently.
The term inference gateway is used in two different ways. In Kubernetes, it usually means infrastructure for routing requests to self-hosted models and choosing the best endpoint (the Gateway API Inference Extension is the reference definition). Elsewhere, people use it to mean a broader control layer for AI traffic, including policy, governance, and observability. Because both meanings are common, we use AI gateway for the broader category and enterprise AI gateway when we mean that larger deployment pattern.
What an AI gateway is not
- Not a model. The gateway does not generate tokens. It decides where a request goes and records what came back. Swapping the gateway does not change the model behind it.
- Not a traditional API gateway. It shares the shape of Kong or Apigee: one chokepoint where governance and telemetry happen. The payload is different. It carries prompts and responses instead of structured REST calls, and its unit of cost is the token.
- Not a security sandbox. The gateway governs the path requests take out of the company. It does not isolate one tool’s execution from another. Runtime isolation is a separate problem.
Concepts in the AI gateway cluster
The cluster follows the journey a platform team actually takes, from understanding the category to operating a deployment:
- Understand: this pillar. The request path, the visibility boundaries, the reference architecture, the taxonomy.
- Architecture: LLM proxy, the data-plane mechanism: interception, protocol compatibility, and how a proxy compares with SDK and OpenTelemetry instrumentation.
- Routing: Semantic and model routing, the decision-signal and destination axes, and What is a semantic gateway, the gateway deployment of content-aware routing.
- Operate: Enterprise AI gateway, the build-vs-buy decision and the rollout path.
- Evaluate: the security, reliability, cost, and deployment scorecard below, applied before and after adoption.
How to evaluate an AI gateway: security, reliability, cost, deployment
Whether a team builds or buys, the evaluation questions are the same four columns. They also make a useful scorecard for a gateway that already exists.
| Dimension | The questions that decide it |
|---|---|
| Security | Where do provider credentials live? What is redacted before it leaves the network, and by what rule? Who can read the capture archive, and is that access itself logged? |
| Reliability | Does streaming pass through faithfully? Do provider errors surface in the shape tools expect? What happens when the gateway itself is down, and is failover to a direct path allowed or blocked? |
| Cost | Is token metering accurate against provider invoices? Can spend be attributed to team, tool, and project? Are budgets enforced at request time or reported after the fact? |
| Deployment | Which provider protocols are supported? Can it run inside your boundary for data-residency requirements? What does leaving it cost: are the archive and the config portable? |
Failure modes when the gateway is missing or half-built
The predictable failures come in two kinds: no gateway, and a gateway that only did the easy part.
- Key sprawl. With no gateway, every tool carries its own provider key. Rotation is manual, spend is a card statement, and an offboarded engineer’s key can outlive them.
- Dark traffic. A prompt from a private repo reaches a public model and the company hears about it from a security report months later, because nothing was on the path to see it.
- The capture-only gateway. A one-line proxy that logs to a flat file handles capture and nothing else: no per-team policy, no queryable archive, no budget enforcement. It works until a second team onboards, then it becomes the system that has to be rebuilt.
- Traffic that skips the gateway. A gateway only governs the traffic routed through it. If tools can still reach providers directly, the records are incomplete in a way that is invisible until an audit needs them. The coverage section above is the antidote.