Portcullis

What it is

Portcullis is a minimal but serious LLM gateway in Go. It sits between applications and OpenAI-compatible providers and gives one control point for the cross-cutting concerns every LLM deployment ends up needing: weighted model routing with provider fallback, per-provider circuit breakers, per-API-key rate limits, a prompt cache, A/B rollouts, and observability.

The dependency set is deliberately tiny: the Go standard library, yaml.v3 for config, and modernc.org/sqlite for the request log. No CGO, no Prometheus client library; the exposition format is hand-rolled because it is just text.

A single YAML file is the whole control plane: providers, model weights, fallback chain, breaker thresholds, rate limits, cache, experiments, and client keys. Unknown keys are rejected and cross-references are validated at load. SIGHUP reloads it live; a failed reload keeps the previous config and is counted in a metric.

Why it matters

Without a gateway, routing and retry policy is smeared across N application codebases, and a degraded provider stalls every request. Portcullis fails fast: each provider has an independent breaker (closed, open, half-open), open candidates are skipped without consuming the request's retry budget, and a 4xx from upstream is passed through verbatim as a client error, never retried, never counted against the breaker.

The cache absorbs duplicate and near-duplicate traffic. Keys are SHA-256 over the canonical request; with normalization on, "Hello World" and "hello world" share an entry. A/B experiments get deterministic sticky assignment per API key via FNV-1a hashing, so model-quality tests need no client changes.

Measured on an M3 Pro in an in-process harness: roughly 21k proxied requests per second and 184k cache-hit requests per second on 12 cores. The proxy number is dominated by the local round trip, so treat the pair as relative evidence: the cache tier is about 9x cheaper than proxying. Tests are race-clean, with table-driven breaker transitions and end-to-end coverage against fake upstreams.

Architecture decisions

Minimal deps as a feature. A gateway is infrastructure you audit before you trust; three dependencies keep the audit surface readable and the binary CGO-free and cross-compilable.

Weights live on models, chain order on providers. An "auto" request picks a model weighted within the first healthy provider; fallback walks the chain. That matches how gateways are actually operated: primary versus fallback, with the model mix configurable.

Experiments override even explicit model requests, because that is what an A/B test of model quality means. The resolved model is always visible in a response header and the request log.

No server WriteTimeout. SSE streams can legitimately outlive any fixed deadline, so per-attempt upstream timeouts and client disconnects bound response lifetime instead. Streaming fallback is pre-headers only; once a stream starts, the gateway completes whatever the provider sends.

The request log is async: SQLite in WAL mode, single writer goroutine, batched transactions. Writes never block the request path; under overload, rows drop and a gauge counts them. If rows are billing-critical, the two-method logger interface fronts Kafka or BigQuery, and a JSONL driver ships for zero extra deps.