One endpoint for every model: designing the AI gateway

Every team that ships AI ends up rebuilding the same plumbing: a client per provider, a retry policy, a cache, a budget guard, and some way to see what actually happened. We pulled all of it into one place. The AI gateway is a single endpoint that sits in front of every model — our own GPUs and external providers alike.

A unified surface

Your app talks to one endpoint with one key. Behind it, the gateway routes to the right model, fails over when a provider degrades, and serves cached responses for repeated prompts so you don't pay twice for the same answer. Swapping models is a config change, not a code change.

Budgets and tracing as first-class features

Every key carries a budget, so a runaway loop can't quietly burn a month's spend. Every request is traced end to end — prompt, model, tokens, latency, cost — so 'why is this expensive?' has an answer you can click on rather than guess at. Cost transparency isn't a dashboard we bolted on; it's metered at the call.

Three tiers of compute

Not every inference needs a datacentre GPU. Small models run client-side on the user's own hardware via WebGPU, mid-size workloads run on our edge, and the heavy jobs go to cloud GPUs. The gateway decides where a call should land so the cheapest capable tier handles it — which is how you make AI cheap enough to put everywhere.