A CISO and a platform engineering lead asked me for the same thing three weeks apart, used the same two words, and wanted completely different products. That is the enterprise AI gateway problem in one sentence. People find Miami Mike looking for a vendor shortlist, and the shortlist is useless until you know which of the two markets you are actually shopping in, because they barely overlap.
The short answer
An AI gateway sits between something and a model, inspects what passes, and enforces policy. The split is what sits on the left side. A developer gateway governs what your applications call. A security gateway governs what your employees and agents do. Most enterprises need both, and one product rarely covers both well.
What is an AI gateway
Gartner defines it as a technology that acts as an intermediary between applications and AI services or models, giving you a central point for security, governance and observability of AI workloads. Mechanically it is a reverse proxy. Requests hit the gateway instead of the provider, the gateway checks credentials, applies rate limits and guardrails, routes to whichever model you configured, and logs the whole exchange on the way back.
The category is young and moving fast. Gartner’s May 2026 market overview found 68% of software engineering leaders already engaged with the technology, with 15% deployed, 31% in the middle of a rollout, and 22% evaluating. The same research projects the market growing from under $250 million in annual revenue to $1 billion by 2028. You can pull the figures from Kong’s reprint of the Gartner report.
AI gateway vs LLM gateway, and why the difference costs money
Here is where procurement gets burned. Two products carry the same label.
The first is infrastructure. It exists because forty teams inside your company are calling OpenAI, Bedrock and an internal endpoint with hardcoded keys and nobody can tell you what any of it costs. Platform engineering buys it. Kong, LiteLLM, Cloudflare, TrueFoundry.
The second is security. It exists because someone in finance pasted a customer list into a free ChatGPT account on their phone. The CISO buys it. Netskope, Zscaler, Cisco, Microsoft.
A proxy that only sees traffic your applications deliberately route through it has no idea the finance incident happened. A network proxy that inspects employee web traffic has no idea what your agents are doing inside a VPC. I have watched a team buy the first one, tell the board shadow AI was handled, and find out otherwise during an audit. Ask any vendor one question before anything else, which is what sits on the left side of their diagram.
What an enterprise AI gateway actually does
Five jobs, in roughly the order buyers care about them rather than the order vendors present them.
- Centralized access. One endpoint, virtual keys scoped per team, provider credentials rotated in one place instead of forty repositories.
- Cost control. Token accounting per user, team, project and agent, with hard budgets that stop requests at the cap. This is the part that gets the project funded, every time.
- Reliability. Failover between providers, load balancing, retries, and caching that serves repeated prompts without calling the model again.
- Observability. Full request and response capture, token counts, latency, error rates, exported to your SIEM.
- Policy. Prompt and response inspection, data loss prevention, access rules by role, and an audit trail you can hand to someone.
Cost is the one worth dwelling on. Finance does not care that a gateway gives you portability. Finance cares that the invoice says Anthropic and nothing in it says which use case burned the money. A gateway is the only thing that answers that question, and if you are budgeting for AI without one you are guessing. I wrote separately about why AI inference costs move differently from the software line items procurement is used to.
The portability argument is real too. A working multi model AI strategy needs somewhere for the routing to live, and building that logic into every application is how you end up unable to change your mind.
The types of AI gateways, sorted by where they sit
Every product in this market is defined by its enforcement point. That single fact tells you more than any feature matrix, because it determines what the thing can physically see.
| Layer | What it sees | What it misses | Typical products |
|---|---|---|---|
| End user device | Clipboard, file operations, USB, native desktop apps, local IDE assistants | Anything on an unmanaged device | Endpoint DLP agents, Microsoft Purview endpoint DLP |
| Browser | What people paste, upload and type into AI tools, personal vs corporate logins | Machine to machine traffic | Island, LayerX, Prisma Access Browser, Chrome Enterprise Premium |
| Network | Broad app discovery, inline inspection of web traffic | Native apps and agents that never touch the proxy | Netskope, Zscaler, Cisco |
| Application and API | Every call your own code and agents make | Employee behavior entirely | Kong, LiteLLM, Portkey, TrueFoundry, Bifrost |
| Edge and cloud | Whatever already routes through that provider | Anything outside that fabric | Cloudflare AI Gateway, Azure API Management, AWS Bedrock |
| MCP and agent | Which tools an agent can discover and execute | The prompt content itself, unless paired with guardrails | Kong, Zscaler MCP broker, Netskope Agentic Broker |
The browser layer is where most of the action is right now, because that is where employees actually use AI. It splits two ways. Island and Prisma Access Browser ship a managed Chromium build that replaces Chrome, which lets them block screenshots and watermark sessions because the vendor owns the browser process. LayerX runs as an extension on whatever browser people already have. The extension approach deploys in an afternoon. The full browser gives you controls the extension cannot reach. Neither is wrong, and you should pick based on how much rip and replace your users will tolerate, which is usually less than the pilot suggests.
A gateway is not only for public APIs, which people forget. If you run models yourself, the gateway is what authenticates and rate limits access to them. That is a real part of the case for on premise AI inference, since owning the hardware only helps if you also own the access path.
Top AI gateway products and brands in 2026
Ten that show up in real enterprise deals. I have worked with some of these and only read documentation on others, and I will say which is which where it matters.
- Palo Alto Prisma AIRS. The most aggressive consolidator in the category. Palo Alto closed its acquisition of Portkey on May 29, 2026 for $117 million, substantially all cash, and made it the AI Gateway inside Prisma AIRS. Combined with Protect AI for model scanning and Talon for the browser, nobody else covers as much surface.
- Cisco AI Defense. Built on the Robust Intelligence acquisition, which shipped one of the first products anyone called an AI firewall. Pairs pre deployment red teaming with runtime inspection and feeds detections into Splunk. The right answer if Cisco Secure Access is already your fabric.
- Netskope Skylight. A full AI security line with a distinct AI Gateway alongside an Agentic Broker, AI Guardrails and red teaming. The Skylight product line claims inline inspection across more than 1,800 AI apps, and it tells a corporate account from a personal login on the same service, which matters more than the raw catalog number.
- Zscaler AI Security Suite. Discovery of models, agents, MCP servers and shadow AI, inline DLP on model calls, and testing mapped to MITRE ATLAS, NIST and the EU AI Act. Strongest where traffic already flows through the Zero Trust Exchange.
- Microsoft Purview and Defender. The default for Microsoft shops, covering Microsoft 365, Copilot, Edge and supported third party AI sites. Read the licensing carefully before you promise anything, because endpoint DLP and the advanced posture features sit behind separate entitlements and some scenarios bill on usage.
- Kong AI Gateway. Now governs three traffic types through one control plane, LLM, MCP and Agent2Agent, with semantic caching, prompt compression, token budgets by consumer group, and an AI Sanitizer policy that redacts PII before the request leaves. It will also proxy AI command line tools like Claude Code, which is a quietly useful thing for anyone trying to govern what developers run locally.
- Cloudflare AI Gateway. The fastest thing on this list to turn on. Caching, rate limiting, analytics and logging with almost no configuration, plus unified billing so third party model usage lands on one Cloudflare invoice.
- LiteLLM. The open source default, and the one I reach for when a customer wants to prove the concept before writing a check. More than 140 providers behind one key, spend tracked per key and team and agent, and it runs air gapped.
- TrueFoundry. LLM, MCP and agent gateway in one platform with deployment tooling underneath. Fits regulated buyers who want the control plane inside their own environment.
- WitnessAI. Purpose built workforce AI governance rather than a platform bolt on, aimed squarely at visibility into what employees and agents are doing.
Also in the conversation depending on the deal: Bifrost, MLflow AI Gateway, Gravitee, Axway, IBM and Apigee on the infrastructure side, and HiddenLayer, Harmonic Security, Lasso, Nightfall and Island on the security side.
The consolidation you need to know about before a vendor call
This market got bought. Cisco took Robust Intelligence in October 2024. Palo Alto took Protect AI in July 2025. SentinelOne took Prompt Security that August. F5 took CalypsoAI and Check Point took Lakera within about two weeks of each other in September 2025. OpenAI took Promptfoo in March 2026. Then Palo Alto took Portkey.
Several products that still get described as independent startups on comparison blogs are now features inside a platform. If a list you are reading has not been updated since last year, half of it is wrong.
Do AI gateways detect shadow AI
The security layer does. The infrastructure layer does not, and this is the single most common misunderstanding I run into.
Discovery comes from CASB catalogs, endpoint agents, browser extensions and proxy logs. It works by recognizing traffic to known AI services and telling you who is using what. Coverage gets quoted as a catalog size, and those numbers vary wildly between vendors because everyone counts differently. Do not compare the numbers. Hand the vendor your actual list of tools in use and ask which ones they see today.
The harder problem nobody solves cleanly is the employee on a personal device with a personal account. If it never touches your network and never touches a managed endpoint, no gateway on the market will find it. That is a policy and culture problem wearing a technology costume.
Does an AI gateway remove PII from prompts
Yes, and with more nuance than the datasheets suggest. The mature pattern is four actions you pick per entity type and per route, rather than one blanket setting.
- Mask, which replaces the value with its type so the model still understands the shape of the sentence
- Hash, when the model needs to know two mentions are the same person without knowing who
- Tokenize against a vault, when the response needs the real value restored on the way back
- Refuse, for anything that must never reach a given provider at all
Detection is layered underneath. Regex handles structured formats like card numbers and account identifiers. A semantic engine such as Microsoft Presidio handles names and free form references that no pattern will catch. A secrets scanner catches the API key someone pasted into a prompt, which happens more than anyone admits. Kong’s approach is documented openly if you want to see what a real implementation looks like, in the Kong AI Gateway documentation.
Three questions separate a real capability from a checkbox. Does it inspect responses as well as prompts, since a model can return a real looking record it pulled from a tool call. Does it scan every streamed chunk rather than only the assembled final answer, because once a token reaches the browser the redaction pass is too late. And does it cover tool call arguments and retrieval context, which is where agentic workflows leak.
Ask where the redaction applies. Some gateways strip data before it reaches the provider. Some leave the live request untouched and only redact what lands in the logs. Those are completely different guarantees and vendors describe both as PII protection.
What an AI gateway does for security and governance
On the security side you get prompt injection and jailbreak detection, blocking of sensitive data leaving, inspection of model output for malicious URLs and unsafe content, access rules scoped by role, and identity for agents so an agent is not logging in with a human’s badge.
Governance is the part that gets undersold, and it is the part public sector buyers care about most. A gateway is the mechanism that turns a written policy into something you can prove. It produces the log that answers a records request. It gives you the retention control you do not get from a consumer account. It enforces the acceptable use rule that otherwise lives in a PDF nobody opens.
That framing lands hard with agencies. When I work through a Florida local government AI policy, the recurring gap is never the framework itself. The framework is fine. The gap is that nothing in the environment can demonstrate the policy is being followed, and a gateway is the cheapest way to close it.
One caution. The gateway log becomes a new repository of sensitive prompt text, sitting somewhere your data classification program has probably never looked. Ask about retention, encryption and who can query it, before you turn logging on and forget about it.
How to choose an enterprise AI gateway
No single layer covers the whole problem. Any vendor telling you otherwise is describing the layer they happen to own.
For most real enterprises an enterprise AI gateway ends up being two purchases that meet in the middle. Platform engineering owns an infrastructure gateway for everything the company builds. Security owns a browser or network control for everything employees use. Trying to make one product do both is where these projects die, usually about four months in, when someone notices the coverage gap and the whole thing goes back to committee.
Gartner’s own commentary on the Portkey deal made a version of this point, warning buyers to manage their gateway strategy independently of the gateways embedded inside larger security platforms, because an embedded gateway gets optimized for the platform’s use case. Palo Alto’s own announcement of the Portkey close is worth reading alongside the purchase price in the PANW annual filing, since the second tells you how seriously to take the first.
What I would not buy is a gateway chosen before anyone inventoried what is actually running. Every organization I have gone through this with underestimated their AI footprint, usually because features arrived inside products they already owned. Buy the inventory first even if the inventory is a spreadsheet and two weeks of proxy logs.
And I have not run any of these at the scale a large agency would need. What I have seen is the procurement conversation go sideways, repeatedly, for the same reason each time.
What I am watching next is MCP. Gartner now lists MCP gateway support as a mandatory feature rather than a differentiator, which is a fast promotion for a protocol that did not exist three years ago. If that holds, the interesting question stops being which model you call and becomes which tools your agents are allowed to touch, and the gateways that only understand prompts are going to look thin.
If you are scoping one of these right now, start by drawing your own version of that layer table and marking which rows you have covered. The empty rows are your answer.
