Definition
An AI gateway is a control point for model traffic: the prompts and responses flowing between a tool and a model provider. An MCP gateway is a control point for tool traffic: the calls flowing between an agent (an MCP client) and the Model Context Protocol servers that expose tools and data. They govern two different planes of the same agent.
Somewhere in the design review for an agent platform, a second gateway appears on the whiteboard. The team already planned an AI gateway for its model traffic. Now a vendor page or a teammate says they also need an MCP gateway, and the two names sound close enough to be the same product.
They are two different products, because an agent does two kinds of work over the network. It talks to models: it sends a prompt and gets back reasoning or text. And it uses tools: it queries a database, calls a search service, restarts a server. The tool side increasingly runs over the Model Context Protocol (MCP), the open protocol agents use to call tools, in which the agent acts as a client and each source of tools, files, or data is an MCP server. Model traffic and tool traffic are two separate planes, and each gateway governs one of them.
An AI gateway sits on the model plane: the prompts an agent sends to a provider and the responses it gets back. An MCP gateway sits on the tool plane: the calls an agent makes to its MCP servers. As agents use more of those servers, the same consolidation pressure that produced the AI gateway for model traffic produces an MCP gateway for tool traffic: one control point instead of many direct connections.
What each gateway sees during one agent incident
An on-call agent works an incident. It reasons about the alert: a model call, through the AI gateway. It queries a metrics MCP server: a tool call, through the MCP gateway. It reasons about the result, another model call, and then restarts a service via an operations MCP server, another tool call.
With both gateways, the platform team can reconstruct the whole incident: the prompts and the model’s reasoning from the AI gateway, and the exact tools invoked, against which servers, with what results, from the MCP gateway. With only the AI gateway, the restart is invisible. The record shows the agent decided to restart a service; it does not show that the restart happened or how it went. The tool plane is where the consequential actions live, which is why leaving it ungoverned is the riskier gap.
An AI gateway sees what the agent asked the model. An MCP gateway sees what the agent asked the tools. Govern one and the other is dark.
How an AI gateway and an MCP gateway differ
| AI gateway | MCP gateway | |
|---|---|---|
| Traffic | Model prompts and responses | MCP tool and resource calls |
| Sits between | A tool and model providers | An agent and MCP servers |
| Core questions | "Which model, what cost, what response?" | "Which server, authorized, what result?" |
| Routing key | Model and provider | MCP server and tool |
| Auth concern | Provider API keys | Server auth, often OAuth |
| Multiplexing | Across providers | Across MCP servers behind one endpoint |
The MCP-gateway capabilities are documented in the open. The Envoy AI Gateway project lists an MCP Gateway that provides tool routing plus server multiplexing, OAuth authentication, and observability for MCP workloads. That a single project documents both an AI gateway and an MCP gateway is itself the shape of the answer: sibling control points, one for each plane.
How an agent uses both gateways in one loop
A capable agent stack runs both, each on its own plane, and the agent uses them in the same loop.
┌──────────── AGENT ────────────┐
│ │
model plane ◄────┤ "reason about the task" ├────► tool plane
│ │ "call the search tool" │ │
▼ └────────────────────────────────┘ ▼
AI GATEWAY MCP GATEWAY
route model, route tool call,
meter tokens, authorize server,
capture prompt/response observe tool result
│ │
▼ ▼
model providers MCP serversThe agent alternates: it asks the model what to do (model plane, AI gateway), then calls a tool to do it (tool plane, MCP gateway), then feeds the result back to the model. Governing both planes is what makes the whole loop observable. Govern only the model plane and you know what the agent thought; the actions are missing. Govern only the tool plane and you know what the agent did; the reasoning and its cost are missing.
Why one gateway cannot govern both model and tool traffic
The two planes ask different questions. The model plane asks which provider served a prompt, what it cost in tokens, and what the response was. The tool plane asks which MCP server handled a call, whether the agent was authorized to use it, and what the tool returned. A gateway built to meter tokens and route by model is not built to authenticate against an MCP server and route a tool call, and vice versa.
What an MCP gateway is not
- Not an AI gateway with a tool tab. It speaks a different protocol (MCP) and governs a different concern (tool access), even when a platform ships both together.
- Not an API gateway. MCP has its own session, transport, and capability model. See AI gateway vs API gateway for the structured-API comparison.
- Not optional once agents use tools. If agents invoke MCP servers, the tool plane exists whether or not you govern it. An MCP gateway is how you stop it being ungoverned.
Failure modes of governing only one plane
- Model-plane-only governance. You capture prompts and cost but not which tools the agent invoked, so the actions with real-world effects are unrecorded.
- Tool-plane-only governance. You see the tool calls but not the model reasoning or its cost, so you cannot explain why the agent did what it did.
- Treating MCP as just another API route. Routing MCP traffic through a generic gateway misses server auth, session semantics, and tool-level observability.
- Two planes, no shared record. If the model and tool records live in unrelated systems, reconstructing an incident that spans both means stitching logs by hand.
When you need an MCP gateway alongside an AI gateway
You need the AI gateway as soon as agents call models at any scale. You need the MCP gateway once agents use MCP tools and the tool plane starts to matter:
- Your agents connect to more than one MCP server and direct connections are becoming unmanageable.
- You need tool calls authorized and observed, in addition to model calls.
- You need to reconstruct what an agent did, which lives on the tool plane, alongside what it thought, which lives on the model plane.
If your agents only call models and use no MCP tools, the MCP gateway is premature. It becomes necessary the moment the tool plane exists and you want it governed.
Concepts related to the AI gateway vs MCP gateway comparison
- AI gateway: the model-traffic control point.
- AI gateway vs API gateway: the structured-API sibling comparison.
- LLM proxy: the primitive on the model plane.