LLM gateways: when you need one, and when you really don't

An LLM gateway is a proxy that speaks one API and talks to many providers. It holds the keys, picks the model, retries when a provider fails, counts what everything cost, and writes it all down. Four or five products do this well, and the question I get asked is which one to buy.

The prior question is whether you need one at all. A gateway is another service on the path of every AI request you serve: its outage is your outage and its latency is your latency. That is a real price, and plenty of teams pay it to solve problems they do not yet have.

I contribute to litellm, including a fix for streaming fallback correctness in its responses bridge, so a provider failing mid-stream degrades instead of dropping the response. That is the sort of detail that decides whether a gateway helps you or quietly hurts you, and it is the lens for this post.

What a gateway actually does

The feature lists across products are close to identical. LiteLLM's proxy describes itself as an OpenAI-compatible server for calling 100+ LLMs with virtual keys, budgets and rate limits, routing, fallbacks, caching, logging, guardrails and spend tracking. Cloudflare's AI Gateway lists caching, spend limits, rate limiting, dynamic routing, guardrails, data loss prevention, bring-your-own-keys, analytics, logging and custom costs. Portkey and OpenRouter cover the same ground with different emphasis.

The differences that matter are operational:

• Where the failover logic lives. LiteLLM's proxy separates plain fallbacks from context_window_fallbacks and content_policy_fallbacks, with num_retries before falling back and allowed_fails plus cooldown_time to take a broken model out of rotation. It also records attempted_fallbacks and original_model_group in the spend logs, so you can tell later which requests were served by the backup.
• Whether failover is provider-level or model-level. OpenRouter load balances across the providers serving a model, "prioritizing price", with allow_fallbacks defaulting to true. That answers a different question from "use Claude, and if Anthropic is down use GPT".
• What happens to streaming. A fallback halfway through a streamed response is the hard case, and it is where correctness bugs live. Ask for the behaviour in writing, then test it by killing a provider mid-stream.

The three shapes

A library runs inside your process. The LiteLLM Python SDK and the Vercel AI SDK both give you one interface over many providers, retries and fallback logic, and a place to hook cost logging. No extra network hop, nothing new to run, and no third party holding your prompts. What you do not get is a key you can hand to another team and revoke, or a central budget someone else cannot bypass.

A self-hosted gateway is the same features as a service you run. LiteLLM and Portkey both publish open-source gateways under MIT. Portkey's repo claims a sub-millisecond overhead and a 122 KB footprint; LiteLLM's README claims 8 ms at the 95th percentile at 1,000 requests a second. Treat vendor benchmarks as directional and measure your own, but the honest summary is that the proxy hop is small next to a model call.

The cost that gets underestimated is operational. Virtual keys in LiteLLM need a database: the docs are explicit that you "need a postgres database (e.g. Supabase, Neon, etc)", a DATABASE_URL and a master key. So you now run a stateful service on the hot path of every AI feature, with upgrades, backups and someone on call.

A hosted gateway hands that work to a vendor. You get dashboards on day one, and you accept that your prompts pass through someone else's infrastructure and that their availability multiplies into yours, in the way every dependency in series does.

What it costs

Two pricing shapes are in use, and the difference compounds with your spend.

OpenRouter's bring-your-own-key route is worth knowing: you can use your own provider keys with no fee up to $25,000 of monthly list-price inference, and 5% above that. Cloudflare's log limits are the constraint on the free path: 100,000 logs in total on the Workers free plan, 10 million per gateway on the paid one.

A percentage fee is fine while spend is small and becomes the most expensive line on the page when it is not. A subscription priced on logged requests does the opposite: it barely moves as spend grows, but a high-traffic, low-cost workload can blow through the log allowance.

When you do not need one

Most teams asking this question have one provider, one application and one team. For them the answer is no.

• One provider, one app. Provider SDKs plus your own logging beat a gateway. Add a library when you want portability, not a service.
• You want fallbacks and nothing else. A retry with a second provider is twenty lines in your own code, and it fails in ways you can debug from your own stack traces.
• You want cost tracking. Log tokens, cached tokens, model and feature per call and put them in your warehouse. You need that log line anyway, and a gateway's dashboard is not a substitute for your own observability.
• You are pre-launch. A gateway solves coordination problems between teams. If there is one team, there is nothing to coordinate.

The strongest genuine reasons to run one are about people, not models: several teams calling models, someone needing per-team budgets and revocable keys, a central place to enforce guardrails, and one dashboard that finance can be pointed at.

Work out your own answer

Tick what is true today, not what might be true next year.

Five things to check before you commit

1. Where the logs live, and for how long. Your prompts are in them. Check retention, region and who at the vendor can read them; Portkey publishes 3-day log retention on its free tier and 30 days on Production, which is also a data-exposure question, not just a debugging one.
2. What happens when the gateway is down. Fail open to direct provider calls, or fail closed? Whichever you choose, write the fallback path first and test it by blocking the gateway's hostname.
3. Streaming and tool calls through a fallback. The common bug is a provider failing mid-stream and the client getting a truncated or empty response rather than a degraded one. Test it deliberately.
4. Whether the cost numbers are yours. Gateways price requests using published rates, which will not match your negotiated rates or your batch discounts. Cloudflare exposes custom costs for exactly this reason. If the dashboard is your source of truth for billing, make sure it is right.
5. How you get out. An OpenAI-compatible interface makes leaving easy; proprietary prompt management and stored routing rules do not. Keep the config in your repo.

None of this argues against gateways. It argues for adopting one when the work it does is work you actually have, and for knowing that the cheapest version of that work is often a library and a good log line. If you want to see how I write that layer, some of it is in the open, and the per-feature cost arithmetic it feeds is in LLM cost per user.

The short version

A gateway is the same set of features wherever you put it. The real question is who operates it and what it costs you to have another service on the request path.

With one team and one provider, use the provider SDKs and a library, and spend the effort on a good log line instead. With several teams, keys to revoke and budgets to enforce, a gateway earns its place, and then the choice is money against staff time: a percentage fee is $250 a month on $5,000 of spend and $2,500 on $50,000, while a self-hosted proxy is about $60 of infrastructure plus a Postgres and someone's attention.

Whichever you pick, test the failure path yourself. The value of a gateway is entirely in what it does on the bad day, and the bugs live in streaming, fallbacks and the moment the proxy itself goes down.