If you’re running multiple LLM providers — OpenAI, Anthropic, Google, local models — you already know the pain. API keys scattered across microservices. No unified view of spend. When one provider goes down, your app goes down with it. And if you’re in a regulated environment, the compliance gaps keep you up at night.

The answer is an AI gateway. And yes, you can build one yourself.

What an AI Gateway Actually Does

An AI gateway is a reverse proxy that sits between your applications and your LLM providers. Standard API gateways like Kong, Nginx, or Envoy handle traditional HTTP traffic, but they don’t understand AI-specific constructs — token budgets, model routing, streaming semantics, or the difference between a $0.50 request and a $5.00 one.

A dedicated AI gateway solves this with a single, standardized endpoint — usually mimicking the OpenAI /v1/chat/completions schema. Your frontend or autonomous agents point to localhost:4000 (or an internal URI), and the gateway handles translating the request to whichever provider you’ve configured.

The core features:

  • Unified API — One endpoint for all providers. Your app doesn’t care whether it’s talking to Claude or GPT-4.
  • Intelligent Routing — Send requests to the right provider based on model name, cost, latency, or availability.
  • Failover and Retry — When OpenAI returns a 429 or 500, the gateway automatically reroutes to a backup provider mid-flight.
  • Rate Limiting — Per-client, per-key throttling to prevent runaway costs.
  • Centralized Key Management — Applications authenticate to the gateway. The gateway holds provider keys. Clients never see them.
  • Caching — Exact-match or semantic caching to avoid paying for duplicate requests.
  • Cost Tracking — Token counting and spend monitoring per project, per client, per model.
  • Audit Logging — Every request, response, token count, and latency metric recorded in one place.

The architecture looks like this:

Your App(s)
    │
    ▼
┌─────────────────────────────────┐
│         AI Gateway              │
│                                 │
│  Auth → Route → Cache → Proxy   │
│    │              │        │    │
│    ▼              ▼        ▼    │
│  Rate Limit   Logging   Failover│
└─────────────────────────────────┘
    │         │         │
    ▼         ▼         ▼
  OpenAI  Anthropic  Ollama (local)

The Two Paths to DIY

The most important lesson from teams who’ve built AI gateways: don’t build the translation layer from scratch. The problem is maintenance. Providers constantly change their APIs — new structured output formats, streaming changes, tool calling updates. Maintaining translations for 50+ providers is a full-time job.

The better approach: deploy a proven open-source proxy and add your own compliance middleware on top.

LiteLLM is the clear leader. It’s a Python-based proxy that translates 100+ LLM APIs into the OpenAI format. You configure it via YAML, deploy it as a server, and it handles routing, load balancing, fallback, and spend tracking out of the box.

# litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: sk-your-key
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: sk-ant-your-key

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  fallbacks:
    - gpt-4o: [claude-sonnet]

One command: litellm --config litellm_config.yaml --port 4000. Your apps point to http://localhost:4000/v1/chat/completions and everything just works.

Portkey Gateway is another solid option — lightweight (~45KB core), TypeScript-based, supports 250+ models, and can be launched with npx @portkey-ai/gateway.

Kong with AI Plugins works well if you’re already running Kong for your API infrastructure. As of Gateway 3.12 (October 2025), Kong ships with native MCP proxy support, OIDC auth, and rate limiting for AI traffic.

Path 2: Build From Scratch (Maximum Control)

If you need highly specialized behavior — intercepting tool-calling loops, enforcing agent constraints, integrating a bespoke token-compression pipeline — you build a custom proxy.

The recommended stacks:

Runtime Best For Why
Python (FastAPI) Compliance middleware Fastest development; rich ecosystem (Presidio for PII, tiktoken for counting)
Node.js (Express) Lightweight proxy Good for JS teams; SSE handling well-understood
Go High-performance edge Minimal latency; single-binary deployment
Rust Enterprise-grade, extreme scale Lowest latency; used by production gateways

A minimal viable gateway is genuinely about 100 lines — a forwarding proxy with model-to-provider mapping. But a production gateway that handles multi-provider streaming, proper error recovery, response caching, and audit logging is realistically 500–2,000 lines.

Here’s the feature difficulty matrix:

Feature Difficulty Notes
Basic Routing Easy Map model name to provider URL
API Key Auth Easy Header validation middleware
Rate Limiting Easy Per-key counters with TTL
Request Logging Easy Append-only log
Exact-Match Caching Medium Hash request body → Redis lookup
Failover/Retry Medium Try-catch with fallback chain
Load Balancing Medium Round-robin or least-busy across keys
Cost Tracking Medium Token counting + per-provider pricing
Semantic Caching Hard Vector embedding + similarity threshold
PII Redaction Hard NER models + regex; de-anonymization on response
Streaming (SSE) Hard Provider-specific formats; pipe-through without buffering
GxP Audit Trail Hard WORM storage, hash chains, 7+ year retention

The Hardest Problem: Streaming

Every team that builds an AI gateway hits the same wall: SSE (Server-Sent Events) streaming.

The issue is that every provider formats streaming events differently. The gateway must pipe tokens through in real-time without buffering the full response — the user sees tokens appear one by one. But if you also need to log the complete response for audit trails (which you absolutely do in GxP environments), you need to concatenate the chunks while simultaneously forwarding them.

This is where simple proxy architectures break down. A naive approach buffers the entire response before forwarding, which destroys the real-time UX. A naive forwarding approach loses the ability to log. The correct implementation uses a TransformStream or equivalent that copies each chunk to a logging buffer while passing it through to the client unchanged.

If you’re building from scratch, budget 30–40% of your engineering time for streaming alone.

What This Means for Regulated Environments

For teams in life sciences, pharma, or any GxP-regulated environment, an AI gateway isn’t optional infrastructure — it’s the only way to deploy AI responsibly.

Why the Gateway Becomes Mandatory

GxP Requirement What the Gateway Does
21 CFR Part 11 §11.10(e) — Audit trails Logs every request/response with timestamps, user identity, model used
21 CFR Part 11 §11.10(d) — Access control Enforces RBAC; clients authenticate to gateway, not directly to providers
PII/PHI Protection Intercepts and redacts sensitive data before it leaves your perimeter
Data Sovereignty Ensures no data is sent to non-approved providers or regions
ALCOA+ Compliance Provides Attributable, Contemporaneous, Original records of all AI interactions

The GxP Compliance Stack

For a regulated DIY gateway, you need these layers in order:

  1. Authentication (OAuth 2.1/OIDC) — Who is making this request?
  2. Authorization (RBAC/FGA) — Are they allowed to use this model?
  3. PII/PHI Redaction (Microsoft Presidio) — Strip sensitive data before it reaches the provider
  4. Prompt Injection Detection — Is this prompt trying to manipulate the system?
  5. Budget/Rate Limit Check — Has this user exceeded their quota?
  6. Cache Check — Can we serve this from cache?
  7. Model Routing + Failover — Which provider handles this request?
  8. Immutable Audit Log (WORM storage, hash-chained) — Record everything

This is the “assembled DIY” approach: deploy LiteLLM as your core proxy, build a FastAPI compliance middleware in front of it, and you have 100% control over governance without maintaining API translations for 50+ providers.

Layer Tool Why
Core proxy LiteLLM (self-hosted) Proven, 100+ providers, OpenAI-compatible
Custom middleware FastAPI Python ecosystem; fast to develop compliance logic
PII/PHI redaction Microsoft Presidio NER + regex; configurable for healthcare data
Audit logging Custom + WORM storage Immutable, SHA-256 hash chained, 7+ year retention
Auth OAuth 2.1 / OIDC Enterprise IdP integration (Entra ID, Okta)
Observability Langfuse (self-hosted) LLM-specific tracing; token cost tracking
Cache Redis Fast, supports exact-match and TTL-based caching
Prompt injection detection Lakera Guard / Rebuff Pre-built detection models

What the Gateway Cannot Do

A critical gap that every team in regulated environments must understand:

  • The gateway secures the pipeline, not the model’s output. If the LLM hallucinates, the gateway cannot catch it. You need a separate output verification layer and human-in-the-loop for high-stakes decisions.
  • The gateway cannot generate e-signatures. It logs actions, but legal signatures require integration with a validated e-signature system.
  • The gateway cannot validate non-deterministic models. Traditional GAMP 5 validation assumes deterministic systems. LLMs are not. You need a separate Computer Software Assurance (CSA) lifecycle for models and prompts.

Security: Non-Negotiable Practices

Practice Why How
Never expose unauthenticated gateway to the internet Anyone could burn through your API keys Behind reverse proxy (Nginx) with auth + HTTPS
Encrypt API keys at rest Gateway stores all provider keys Secrets manager (Vault, AWS Secrets Manager)
Network isolation Gateway should be in private subnet VPC deployment, no public IP
Key rotation Stale keys are a security risk Automate rotation; short-lived tokens where possible
Redact keys in logs Keys can appear in debug output Strip authorization headers from all log output

The Bottom Line

Building a DIY AI gateway is one of the most practical infrastructure investments you can make in the AI stack. The architecture is well-understood, the open-source tooling is mature, and the path is clear.

For most teams: deploy LiteLLM as your core proxy, add a custom compliance middleware layer, and point your apps to localhost:4000. That gives you enterprise-grade AI infrastructure with full control over governance, cost, and data sovereignty.

For regulated environments: add Presidio for PII redaction, Langfuse for observability, and WORM storage for audit trails. The gateway becomes the backbone of your GxP AI strategy — the single enforcement point between your data and the outside world.

Budget real engineering time. A basic setup takes 1–2 weeks. A compliance-ready regulated deployment takes 1–2 months. And if you’re building from scratch, expect 500–2,000 lines of code for a production-quality gateway — not the “100 lines” that gets thrown around for a minimal forwarding proxy.

The tools are there. The architecture is proven. The only question is whether you build it now, or after the first compliance audit forces you to.