LLM gateway explained: what it is and why you need one
An LLM gateway is a single API endpoint fronting multiple model providers. How it works, why enterprises adopt one, and what to look for.
by Elias Mahdavi · Published on · 15 min read
An LLM gateway is a single API endpoint that sits between applications and multiple AI model providers, handling authentication, routing, per-user keys and usage metering in one place. LiteLLM is the leading open-source option, named directly in Google's AI Overview for this query; DevKira ships it natively as a core platform capability.
Key takeaways
- An LLM gateway is a single OpenAI-compatible API endpoint that fronts multiple model providers, so applications call one interface instead of integrating each provider's SDK separately.
- LiteLLM is the most widely cited open-source LLM gateway, and DevKira ships it natively rather than as a bolt-on integration.
- A gateway issues one virtual key per user, so every request can be authenticated, budgeted and attributed to a named individual rather than a shared team API key.
- Request-level metering turns AI spend from a monthly surprise into a line-by-line ledger, broken down by person, model and project.
- An MCP gateway and an LLM gateway solve related but distinct problems: one governs which models can be called, the other governs which tools and data an agent can reach.
- Enterprise buyers evaluating an LLM gateway should check for multi-provider routing, per-user authentication, spend limits and audit logging before anything else.
Contents
- What is an LLM gateway?
- What is the most popular LLM gateway?
- 8 reasons enterprises adopt an LLM gateway
- LLM gateway vs LLM proxy vs API aggregator
- What is the difference between an MCP gateway and an LLM gateway?
- What to look for in an enterprise LLM gateway
- LLM gateways, audit trails and UK/EU compliance
- Frequently asked questions
- Getting started with a governed LLM gateway
What is an LLM gateway?
An LLM gateway is infrastructure that sits between an organisation's applications and the AI model providers it uses, exposing one consistent, OpenAI-compatible API regardless of which model actually answers the request. Instead of a developer wiring a separate SDK for each provider, every call goes to the same endpoint, and the gateway decides where it goes next.
That single point of entry is what makes governance possible. A request arrives with a virtual key rather than a raw provider credential, the gateway checks who the key belongs to, what they are allowed to spend, and which model they are permitted to reach, then forwards the call and logs the outcome. Nothing about the calling application needs to change when a new model is added, swapped, or retired behind the gateway.
Why this matters more than it sounds
Most organisations did not choose to end up with model access scattered across five teams and five sets of raw API keys; it happened gradually as each team adopted whichever model solved its immediate problem. An LLM gateway is the point at which that sprawl gets pulled back into one governed layer, without asking every team to rewrite how they call a model.
What "OpenAI-compatible" actually means in practice
An OpenAI-compatible endpoint means the gateway accepts requests in the same JSON shape as OpenAI's chat completions API, the format most SDKs, frameworks and internal tools were already built against. A team can point an existing integration at the gateway URL, swap the model name in the request, and keep every other line of code unchanged, whether the request is ultimately served by Claude, GPT or a self-hosted open-weight model.
That compatibility is what makes self-hosting a gateway realistic rather than theoretical. Because LiteLLM runs inside DevKira's own environment rather than as an external hosted service, no request or response ever needs to leave the workspace just to be routed; the routing decision, the authentication check and the metering event all happen on infrastructure the organisation already controls.
What is the most popular LLM gateway?
LiteLLM is the LLM gateway most frequently cited as the default open-source choice, including by Google's own AI Overview for this exact query. It presents an OpenAI-compatible interface in front of well over a hundred model providers, which is why DevKira ships it natively rather than building a proprietary routing layer from scratch.
The wider landscape includes several credible alternatives with different trade-offs. OpenRouter operates as a hosted, multi-model API marketplace rather than self-hosted infrastructure; TrueFoundry positions itself as a broader AI-platform product with gateway functionality bundled in; and llmgateway.io focuses specifically on the gateway layer as a standalone hosted service. None of these currently has the incumbent-scale brand recognition of a hyperscaler product, which is itself notable: this is a technical category still being defined by open tooling rather than settled by a single vendor.
DevKira's choice of LiteLLM reflects the same logic that makes a gateway worth adopting in the first place: an open, widely adopted routing layer avoids trading dependency on five model providers for dependency on one gateway vendor.
Whether a gateway is hosted externally or run inside the organisation's own environment is not a minor implementation detail. A hosted gateway is another third party that sees every prompt in transit, which reopens the exact data-sovereignty question a governance programme is trying to close. Running LiteLLM natively inside the workspace, as DevKira does, keeps that routing decision, and the traffic it routes, inside the perimeter the organisation already controls, rather than adding a new external dependency in the name of solving the last one.
8 reasons enterprises adopt an LLM gateway
The case for a gateway is rarely made on a single feature. It tends to be a combination of cost visibility, security posture and developer experience that individually might be tolerable, but together push a growing AI footprint past what ad hoc API keys can handle safely.
1. Vendor flexibility without rewriting application code
Because every call passes through the same OpenAI-compatible endpoint, switching the underlying model, adding a new provider, or running two providers side by side for a migration does not require touching application code. The routing decision lives in the gateway configuration, not scattered across every service that calls a model. When a better or cheaper model becomes available, the change happens once, centrally, instead of once per integration.
2. Per-user accountability instead of shared API keys
A shared team API key makes every request indistinguishable from every other. A gateway issuing one virtual key per person means a spike, a misuse, or a policy violation can be traced to an individual rather than investigated across an entire team's activity.
3. Real-time cost attribution and budget enforcement
Token spend becomes visible per person, per model and per project as requests happen, rather than surfacing as one undifferentiated number on next month's provider invoice. Budgets attached to a virtual key can cap spend before it happens, not report on it afterwards, which is the difference between a finance team forecasting AI cost and a finance team discovering it.
4. Centralised audit logging for every prompt and response
Every request passing through one endpoint means every request can be logged in one place: who called which model, when, and with what outcome. That log is the evidence an auditor or a regulator asks for when the question is "how do you know your AI policy is actually followed."
5. Resilience through provider failover
If a model provider has an outage or degrades, a gateway can route around it to an equivalent model without the calling application needing to know that happened. Multi-provider routing turns a single point of failure into a redundant one.
6. Consistent data-handling policy across every model call
A gateway is the one place a data-sovereignty rule can be enforced with certainty: which requests are permitted to reach a given provider, and which must be redirected to an EU-sovereign or on-premise model instead. Enforcing that rule five times, once per team's integration, is how it eventually gets missed once, and the one time it is missed is the time that matters.
7. Faster developer onboarding
A new engineer needs to learn one API shape and request one virtual key, not five separate provider accounts with five separate credential-management processes. That is a meaningful reduction in the time between a new hire's first day and their first productive commit.
8. A single point to enforce which model tier is appropriate
Not every task needs the most expensive frontier model, and not every task is safe to send to any model at all. A gateway is the natural place to route by task sensitivity and cost tier at once, rather than leaving that judgement to individual habit. A drafting task and a task touching regulated customer data should never be one routing decision away from each other; a gateway makes that separation structural rather than aspirational.
LLM gateway vs LLM proxy vs API aggregator
These three terms get used loosely and interchangeably in vendor marketing, which makes evaluating products harder than it should be. The table below separates what each term actually implies.
| Term | Core function | Typical scope | Example use |
|---|---|---|---|
| LLM gateway | Authentication, routing, metering and policy enforcement across multiple providers | Enterprise-wide, governance-first | One virtual key per employee, budgets, audit logs, sovereignty routing |
| LLM proxy | Forwards requests to a model provider, often with light transformation | Narrower; frequently a single provider or a caching layer | Rewriting requests to fit a provider's schema, adding a cache in front of one API |
| API aggregator | Presents many third-party APIs (not only AI models) through one marketplace interface | Broad, discovery-oriented, usually hosted rather than self-run | Comparing and paying for multiple providers through one billing relationship |
In practice, a serious LLM gateway does everything a proxy does and adds the governance layer on top: authentication per user, not just per app; spend limits, not just request forwarding; and an audit trail suitable for a compliance review, not just server logs. "LLM proxy" is frequently used as a simpler synonym for the same category, but the term alone says nothing about whether governance controls exist behind it.
What is the difference between an MCP gateway and an LLM gateway?
An LLM gateway governs which language models an application or person is allowed to call; an MCP gateway governs which tools, files and external actions an AI agent is allowed to reach once it is already running. They sit at different layers of the same governance problem and are increasingly deployed together rather than as alternatives.
The Model Context Protocol standardises how an agent connects to tools, databases and services beyond the model itself, so an MCP gateway is really an access-control layer for that tool-calling surface: which data source can this agent query, which action can it take, under whose authorisation. An LLM gateway never makes that decision; it only decides which model answers a prompt and logs that it did.
DevKira addresses both layers without treating them as separate products bolted together. LiteLLM handles the model-access layer natively, while governed egress, an authenticated forward proxy with an explicit allow-list and deny-by-default posture, controls what any tool, script or agent running inside the workspace can actually reach on the network. The two layers are related but answer different questions, and a mature AI governance programme needs a clear answer to both, not just the model-routing one.
What to look for in an enterprise LLM gateway
Not every product marketed as a gateway offers the same depth of control. The criteria below are the ones that separate a genuine governance layer from a thin routing wrapper.
| Criterion | Why it matters |
|---|---|
| OpenAI-compatible endpoint | Existing tooling and SDKs work without a rewrite; this is the interface LiteLLM exposes and the one DevKira standardises on |
| Per-user virtual keys | Every request traces to a named person, not a shared team credential |
| Multi-provider routing | Claude, GPT and other providers reachable through one endpoint, with the ability to route by task or sensitivity |
| Spend limits per key | Budgets are enforced before overspend happens, not reported after the invoice arrives |
| Request-level metering | Usage attributed by company, by model and by individual, not aggregated into one opaque total |
| Audit logging | A record suitable for an internal review or an external audit, not just raw server logs |
| Data-sovereignty routing | The ability to direct sensitive requests to a masked-cloud, EU-sovereign or fully on-premise model rather than one fixed provider for everyone |
DevKira's gateway is built to satisfy all seven at once, because a gateway that only handles routing without the governance controls around it solves the integration problem and leaves the accountability problem exactly where it started. Cost metering and virtual keys exist alongside the routing layer, not as a separate product a team has to bolt on afterwards.
LLM gateways, audit trails and UK/EU compliance
An LLM gateway's request log is not just an operational convenience; it is increasingly the evidence base compliance frameworks expect an organisation to produce. Under the EU AI Act (Regulation (EU) 2024/1689), providers and deployers of higher-risk AI systems face explicit record-keeping and logging obligations, and a centralised gateway is the most direct way to generate that record automatically rather than reconstructing it after the fact, a point the European Commission sets out on its <a href="https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai" target="_blank" rel="noopener">regulatory framework for AI policy page</a>.
For UK organisations, the relevant guidance sits closer to home. The National Cyber Security Centre's <a href="https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development" target="_blank" rel="noopener">Guidelines for Secure AI System Development</a> treats security as a requirement across the entire lifecycle of an AI system, not just at build time, which is exactly the argument for putting authentication, logging and access control in a gateway rather than leaving each integration to implement its own version. The Information Commissioner's Office is equally direct in its <a href="https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/" target="_blank" rel="noopener">guidance on AI and data protection</a>: accountability under UK GDPR means being able to demonstrate what a system did with personal data, which is difficult to demonstrate at all without a request-level log.
There is also a routing dimension to compliance that a gateway is uniquely placed to enforce: not every request should go to the same model. DevKira's three data-sovereignty postures, masked cloud, EU-sovereign, and on-premise, are enforced at the gateway layer, so the decision about where a given class of data is permitted to go is made once, centrally, rather than trusted to individual judgement on every call. The full mechanics of that jurisdictional question are covered in <a href="/en/blog/data-sovereignty-meaning-explained">what data sovereignty actually means for AI workloads</a>.
Finally, NIS2 (Directive (EU) 2022/2555) matters to UK firms indirectly but concretely: any UK company supplying into an EU customer's essential or important-sector supply chain can expect security requirements, including how AI providers are used and audited, to flow down contractually even without the directive applying to the UK company directly. The EU Agency for Cybersecurity, <a href="https://www.enisa.europa.eu/" target="_blank" rel="noopener">ENISA</a>, frames exactly this kind of third-party AI dependency as a supply-chain risk to be actively managed, not merely disclosed. A gateway that already produces a clean, per-provider audit trail is the difference between answering that request in an afternoon and reconstructing months of usage from scattered logs.
Frequently asked questions
What is an LLM gateway?
An LLM gateway is a single API endpoint that sits between applications and multiple AI model providers, handling authentication, routing, per-user keys, spend limits and usage logging. It lets an organisation call Claude, GPT or other models through one consistent interface instead of integrating each provider separately.
What is the most popular LLM gateway?
LiteLLM is currently the LLM gateway most widely cited as the default open-source option, including in Google's own AI Overview for this query. It offers an OpenAI-compatible interface in front of a large number of model providers, which is why DevKira ships it as a native platform capability.
What is the difference between an MCP gateway and an LLM gateway?
An LLM gateway controls which language models a request can reach; an MCP gateway controls which tools, files and external actions an AI agent can use once it is running. They operate at different layers of AI governance and are commonly deployed alongside each other rather than as substitutes.
Is an LLM gateway the same thing as an API gateway?
No. A general API gateway routes and secures calls to any backend service. An LLM gateway is a specialised version built around the specific needs of model calls: per-token cost metering, provider failover for models specifically, and prompt-level audit logging that a generic API gateway does not provide out of the box.
Do I need an LLM gateway if my company only uses one model provider?
Even with a single provider today, a gateway still delivers per-user accountability, budget enforcement and audit logging that a raw API key does not. Most organisations that start with one provider add a second within a year, at which point the gateway is already in place rather than a retrofit under pressure.
How does an LLM gateway help with GDPR or UK GDPR compliance?
By producing a request-level log of who called which model, with what data, and when. That log is the practical evidence an accountability obligation under UK GDPR requires; without it, demonstrating how personal data was processed by an AI system depends on reconstructing scattered, incomplete records after the fact.
Getting started with a governed LLM gateway
The pattern is consistent across every organisation past a certain size: AI adoption starts with individual teams calling a model directly, and the gateway conversation starts the moment someone asks a question nobody can answer, usually "what did we spend on AI last month" or "which team is calling which model with what data." DevKira's LiteLLM-based gateway, with one virtual key per person, per-request metering and sovereignty-aware routing, is built to have that question answered before it gets asked twice.
The same governance logic extends beyond model access to policy and audit ownership more broadly, covered in <a href="/en/blog/what-is-ai-governance">what AI governance actually means in practice</a>, and to the cost side specifically in <a href="/en/blog/controlling-ai-costs">how to attribute and control AI spend per user and per model</a>. Teams still operating on shadow, ungoverned model access, the problem a gateway is built to close, are covered in <a href="/en/blog/what-is-shadow-ai">what shadow AI is and how to bring it under control</a>.
To see the gateway, the metering and the sovereignty routing working together in one workspace, <a href="/en/book-a-demo">book a DevKira demo</a>.
