LiteLLM vs Portkey vs Kong vs Cloudflare AI Gateway

Q: What is an AI gateway and do I actually need one?

An AI gateway (also called an LLM gateway or LLM proxy) is a control plane that sits between your applications and the LLM providers they call, centralizing the concerns that otherwise get duplicated across every service: routing to the right model, failover between providers, retries, rate limiting, per-team budgets and virtual API keys, semantic caching, input/output guardrails, and observability of tokens, cost, latency, and errors. You need one the moment your setup crosses any of four thresholds: more than one model in use, more than one provider, a spend that finance has started asking about, or a security review that wants to know where prompts and completions go. Below those thresholds a single SDK call in one service is fine. Above them, the alternative to a gateway is the same logic — keys, retries, budgets, logging — copy-pasted into every service, with no central place to rotate a leaked key, cap a runaway agent, or see total cost. That sprawl is exactly the failure the gateway exists to prevent. The gateway does not make your models better; it makes the operation of many model calls governable, which is a platform concern rather than an application one. Whether you buy a managed gateway or self-host an open-source one, the value is the single, enforceable point of control it gives you over traffic that would otherwise be scattered and ungoverned.

Q: Should I choose an AI gateway on the number of models it supports?

No — provider and model count is the least useful criterion, even though it is the one vendors lead with. The headline numbers ("100+ providers", "1,600+ LLMs") are marketing surface area; in practice a given product uses a handful of providers, typically the two or three frontier labs plus maybe an open-weight model and an internal one. Any serious gateway already covers all of those, so the difference between 100 and 1,600 supported models almost never changes which one is right for you. What actually decides fit is the deployment and ownership model: whether the gateway can run inside your own network or is managed-only, whether it fits a stack you already operate, and which capabilities sit in the free tier versus a paid one. A team that must keep prompts inside its VPC for compliance cannot use a managed-only gateway no matter how many models it lists. A team already running Kong for its APIs gets more value from Kong AI Gateway than from a longer model list elsewhere. A startup that wants one command to get started values Cloudflare's one-line managed setup or LiteLLM's pip-installable proxy over coverage breadth. Treat model count as table stakes — confirm your five providers are supported, which they will be — and then ignore it, spending your evaluation on deployment model, latency posture, the OSS-versus-paid split, and how the gateway fits the infrastructure and skills you already have.

Q: What is the real difference between LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway?

The defining difference is deployment model, and it separates them cleanly. LiteLLM is an open-source (MIT) LLM proxy you self-host — a pip-installable or Docker proxy exposing 100+ providers behind one OpenAI-compatible API, with virtual keys and per-key budgets in the free tier and an Enterprise tier adding SSO, RBAC, and audit logs priced to scale rather than per token. Kong AI Gateway extends Kong Gateway with AI plugins and is aimed squarely at organizations that already run Kong: it supports every deployment mode including Kubernetes, but its routing and transformer plugins are open source while the production-grade ai-semantic-cache and ai-rate-limiting-advanced plugins require Konnect or Enterprise. Cloudflare AI Gateway is fully managed on Cloudflare's edge — one line of code to start, available on the free tier, with caching, retries, fallback, guardrails, and analytics, but no self-host option at all, which rules it out wherever data cannot transit a third party. Portkey occupies the hedge position: an MIT-licensed gateway core you can self-host, plus a hosted control plane and enterprise path, with 50+ guardrails and a claimed sub-millisecond internal processing overhead on the self-hosted core. So the four are not interchangeable feature sets; they are four points on a spectrum from fully self-hosted to fully managed, and where your compliance boundary and your existing stack fall on that spectrum picks the tool.

Q: Can I self-host these AI gateways, or are they SaaS only?

It depends on the product, and this is the single most important thing to check before evaluating features. LiteLLM is self-host-first: it is an open-source MIT-licensed proxy you run yourself via pip or Docker, entirely inside your own network, with a separate Enterprise tier that adds governance features but keeps the same self-hosted core. Kong AI Gateway is also self-hostable — it is built on Kong Gateway and supports all deployment modes, including traditional self-managed, hybrid, DB-less, and Kubernetes via the Kong Ingress Controller — though some AI plugins are gated behind Konnect or Enterprise. Portkey offers both: its gateway core is open source (MIT) and can be self-hosted, and it also provides a hosted control plane and enterprise private-cloud options, so you can start managed and move in-house or vice versa. Cloudflare AI Gateway is the exception — it is fully managed SaaS only, running on Cloudflare's edge network with no self-host option. That distinction is decisive for anyone with data-residency or data-boundary requirements: if prompts and completions are not permitted to transit a third-party service, Cloudflare and any other managed-only gateway are eliminated regardless of their features, and your shortlist collapses to the self-hostable options — LiteLLM, Kong, or the Portkey core. Conversely, if managed is acceptable, the fully-managed options remove the operational burden of running the gateway yourself, which is itself a real cost.

Q: Which AI gateway is best if I already use Kong for my APIs?

If you already run Kong Gateway for your traditional APIs, Kong AI Gateway is very likely your best choice, and for a specific reason: it lands the AI control plane on infrastructure your platform team already operates, understands, and monitors, rather than introducing a second system to run. Kong AI Gateway is not a separate product but a set of AI plugins extending Kong Gateway — ai-proxy and ai-proxy-advanced for multi-provider routing and load balancing, ai-request-transformer and ai-response-transformer, ai-semantic-cache, ai-rate-limiting-advanced, and ai-guardrails — so your AI traffic flows through the same gateway, the same policies, the same observability, and the same deployment topology (including Kubernetes via the Ingress Controller) as the rest of your API estate. That consolidation is the value: one control plane, one operational model, one place for auth, rate limiting, and data-security policy across both classic APIs and LLM calls. The caveat to budget for is tiering: the core AI plugins are free and open source, but the production-grade ai-semantic-cache and ai-rate-limiting-advanced plugins, plus the GUI and analytics, require Konnect or Enterprise. The inverse also holds and is an anti-pattern worth naming: if you do not already run Kong, adopting Kong AI Gateway solely for its AI plugins means inheriting Kong's full operational surface for a fraction of its value, and a lighter option like LiteLLM or Portkey will usually serve you better.

Q: Do AI gateways add latency to LLM calls, and how much?

Yes, a gateway is an extra hop on every model call, so it always adds some latency — but how much depends entirely on where the gateway sits, and vendor headline numbers are easy to misread. A self-hosted gateway running inside your own network adds only its internal processing time plus the negligible in-network hop: Portkey's advertised sub-millisecond figure (its "<1ms" claim, with a 122kb footprint) is exactly this — the internal overhead of the self-hosted gateway core, not an end-to-end measurement. That is genuinely small relative to the hundreds of milliseconds an LLM inference call itself takes. A managed edge gateway is different: your request now travels to the provider's edge plane and back, so the added latency is a real network round trip. Portkey's own documentation cites roughly 20 to 40 milliseconds for its hosted product, and any managed gateway will add a comparable network cost depending on geography. Cloudflare AI Gateway mitigates this by running on a global edge network close to users, but it is still a managed hop rather than an in-VPC one. The practical rule is to measure the latency of the exact deployment you will ship, from your application, rather than quoting the vendor's best-case internal number to stakeholders — a self-hosted proxy and a managed edge service have materially different latency profiles, and conflating the two leads to unpleasant surprises in production. In most systems the gateway's added latency is dwarfed by inference time, but for latency-critical paths the self-host-versus-managed choice is worth measuring explicitly.

Q: Are the advanced features of these gateways free, or behind a paid tier?

Many of the features that make a gateway production-worthy sit behind a paid tier, and mistaking "supported" for "free" is a common budgeting error. The pattern differs by product. LiteLLM puts a lot in its open-source tier — routing, fallbacks, virtual keys, per-key and per-team budgets, spend tracking, and logging are all free — while reserving SSO, SCIM, RBAC, audit logs, and enterprise support for LiteLLM Enterprise, which is priced to gateway scale rather than per token. Kong AI Gateway open-sources its core AI plugins (AI Proxy, the request and response transformers, and the prompt guard, template, and decorator plugins), but the two plugins most associated with production cost-control and reliability — ai-semantic-cache and ai-rate-limiting-advanced — plus the management GUI and analytics require Konnect or Enterprise. Portkey's gateway core is open source under MIT and includes fallbacks, retries, load balancing, and 50+ guardrails, with usage-based hosted tiers and enterprise for the managed control plane and advanced governance. Cloudflare AI Gateway is available on all plans including the free tier, with usage-based pricing as you scale into higher volumes and features like persistent logs. The takeaway for evaluation is to map the specific features you need — semantic caching, advanced rate limiting, RBAC, SSO, guardrails — against each product's free-versus-paid line, and price the tier you will actually run in production, because a compelling open-source demo can quietly require a paid tier for the exact capability that justified the gateway.

Q: What is the biggest mistake teams make choosing an AI gateway?

The biggest mistake is choosing on the feature grid instead of the deployment and ownership model, because features converge and deployment model does not. Teams line up spec sheets, compare model counts and capability checkmarks, and pick the longest list — but every serious gateway already routes, caches, rate-limits, applies guardrails, and emits observability, and whatever gap exists today closes within a quarter. What does not converge, and what you will live with for years, is who runs the gateway and where it sits: a fully-managed edge product and a self-hosted OSS proxy are structurally different commitments with different latency profiles, different compliance stories, and different operational burdens, and no feature comparison surfaces that difference. The concrete failure modes that follow from ignoring it are predictable. Choosing a managed-only gateway in a data-residency regime that forbids prompts transiting a third party is a compliance violation dressed up as convenience. Self-hosting a control plane your team will not actually patch, scale, and monitor turns the gateway into a single point of failure on every model call — worse than a managed service you do not operate. Adopting Kong AI Gateway without already running Kong means carrying Kong's whole operational surface for a sliver of its value. And quoting a vendor's best-case internal latency to stakeholders when you will ship the managed round trip sets expectations you cannot meet. The fix is to decide the ownership model first — must-self-host or managed-fine, existing-stack fit, library-or-product — and let that collapse the field before you ever open the feature comparison.

Satyam Kumar

返回博客

ai-engineering

LiteLLM vs Portkey vs Kong vs Cloudflare: The AI Gateway You Actually Need (2026)

By Satyam KumarAugust 31, 202611 min read

ai gateway llm gateway litellm portkey kong ai gateway cloudflare ai gateway llm proxy llm routing ai gateway comparison llm observability semantic caching ai guardrails multi-provider failover llm cost governance ai platform engineering llmops

LiteLLM vs Portkey vs Kong vs Cloudflare: The AI Gateway You Actually Need (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

LiteLLM vs Portkey vs Kong vs Cloudflare: The AI Gateway You Actually Need (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment