Choosing an AI gateway is one of the most consequential infrastructure decisions for teams deploying LLM applications at scale. The gateway sits in the critical path of every request, so its performance directly impacts user experience, cost, and reliability.
We ran a controlled benchmark comparing seven AI gateways — Aurora, Kong, APISIX, Bifrost, Helicone, Portkey, and LiteLLM — on identical hardware under identical load. Every gateway was configured for minimal overhead per its own documentation, with usage tracking, request logging, and non-essential features disabled. The results reveal significant differences in throughput, latency, and reliability.
Benchmark Setup
All gateways were tested on a consumer-grade laptop with an Intel Core i5-10300H processor (4 cores, 8 threads at 2.5 GHz) and 8 GB of DDR4 RAM running Windows 10. While not representative of production server hardware, this setup provides a level playing field for relative comparison and highlights architectural efficiency differences.
Each gateway received rate-limited load at 5,000 requests per second for 60 seconds, forwarded via HTTP/1.1 to a shared mock OpenAI-compatible backend running on localhost. The mock server returned pre-encoded chat completion responses to eliminate provider latency as a variable. A warmup phase of 25 requests plus 256 prewarm payloads was used before each measurement.
The load generator used QPC (Query Performance Counter) timing on Windows and maintained an exact 5,000 req/s pacing. Each gateway was tested in sequence (not concurrently) to avoid port contention and resource competition.
Detailed Results
Aurora achieved the highest throughput at 4,986 requests per second — 99.7% of the 5,000 target — with zero errors across 299,182 total requests. Only Kong and APISIX came close, each exceeding 4,700 req/s with sub-100ms P99 latency. The remaining gateways fell significantly behind.
Gateway Requests Throughput Success P50 P99 P999 Aurora 299,182 4,986/s 100.00% 17.7 ms 47.2 ms 97.2 ms Kong 289,964 4,833/s 99.87% 39.2 ms 98.0 ms 145.6 ms APISIX 286,685 4,778/s 99.86% 39.6 ms 109.1 ms 185.7 ms Bifrost 178,182 2,970/s 99.79% 75.4 ms 285.8 ms 432.9 ms Helicone 143,881 2,398/s 99.68% 102.0 ms 217.3 ms 280.8 ms Portkey 58,651 978/s 99.40% 243.0 ms 480.6 ms 818.6 ms LiteLLM 6,992 117/s 56.74% 1.6 s 24.2 s 30.0 s
Throughput (req/s)
P99 Latency (ms) — lower is better
Memory and Allocation Efficiency
Beyond throughput and latency, allocation efficiency predicts how a gateway behaves under sustained load. Fewer allocations mean less garbage collection pressure and more stable latency over hours-long production runs.
Gateway Allocs/op Bytes/op Aurora 72.2 5,945 APISIX 72.2 5,944 Kong 80.0 6,505 Portkey 87.1 6,812 Helicone 91.1 11,043 Bifrost 92.0 6,960 LiteLLM 103.2 8,533
Aurora and APISIX tie for the lowest allocation count at 72 allocs per request. Kong uses slightly more at 80 allocs/op. Helicone stands out with 11,043 bytes/op — nearly double Aurora — due to Node.js runtime overhead for object allocation.
Throughput Analysis
The three compiled-language gateways — Aurora (Go), Kong (Lua on Nginx C core), and APISIX (Lua on Nginx C core) — all exceeded 95% of the 5,000 req/s target. This demonstrates that well-optimized compiled runtimes can sustain high throughput on modest hardware.
Bifrost (Go with tokio async runtime) reached only 59% of the target at 2,970 req/s. In a separate three-gateway comparison, Bifrost performed better at 4,501 req/s when tested earlier in the sequence before LiteLLM's port contention affected subsequent runs. Bifrost's advertised 5,000+ req/s performance was measured on Linux server hardware with 4 GB RAM per core.
Node.js-based gateways (Helicone at 2,398 req/s, Portkey at 978 req/s) suffer from garbage collection pauses under sustained load. Portkey's particularly low throughput is explained by its heavy middleware stack and per-request object allocations. Python-based LiteLLM collapses entirely at 117 req/s with only 56.74% success rate — the Global Interpreter Lock prevents true concurrent request processing.
Latency Deep Dive
P50 latency tells the story of median request processing overhead. Aurora's 17.7 ms is more than 2.2x faster than the next-best gateway (Kong at 39.2 ms) and 4.3x faster than Bifrost at 75.4 ms.
P99 latency is where architectural differences become stark. Aurora maintains 47.2 ms at the 99th percentile — this means 99% of requests complete within 47.2 ms even under sustained 5,000 req/s load. Kong (98.0 ms) and APISIX (109.1 ms) show competitive but higher tail latency.
P999 latency (the worst 0.1% of requests) further separates the gateways. Aurora's 97.2 ms P999 means even the most delayed requests complete in under 100 ms. Bifrost's 432.9 ms P999 means 1 in 1,000 requests waits nearly half a second. LiteLLM's P999 of 30 seconds makes it unusable for any latency-sensitive application.
Reliability Under Load
Aurora was the only gateway to achieve 100% success rate across the full 60-second run. All other gateways experienced timed-out requests under sustained load, with error rates ranging from 0.13% (Kong) to 43.26% (LiteLLM).
The errors were exclusively context deadline exceeded and connection refused — no gateway produced corrupt responses or incorrect data. This confirms that the failures are performance-driven, not correctness issues.
Configuration Impact
A separate comparison measured the impact of Aurora's configuration options on throughput. Three modes were tested:
- Default configuration: Usage tracking and request logging enabled (previous defaults). Achieved 4,804 req/s at 5,000 target.
- Optimized configuration: Usage tracking and request logging disabled (current defaults). Achieved 4,811 req/s.
- Bench mode: All background features disabled, h2c upstream. Achieved 4,900 req/s — the configuration used in the seven-gateway comparison.
The performance difference between optimized configuration and bench mode is less than 2%, confirming that Aurora's runtime overhead is dominated by its core request processing pipeline, not optional features.
Methodology Notes
Each gateway was tested independently in sequence. The test order was: Aurora first (no prior port contention), then Kong, then APISIX, then Bifrost, then Helicone, then LiteLLM, then Portkey. Gateways tested after LiteLLM may show degraded results because LiteLLM's process blocks its port for 60+ seconds after shutdown.
All gateways were prewarmed with 25 requests plus 256 prewarm payloads before measurement began. The mock server was verified to handle 10,000+ req/s to ensure it was not the bottleneck.
Key Takeaways
- Runtime language choice is the dominant factor in gateway throughput — compiled languages (Go, C-based) outperform interpreted runtimes by 10-40x at 5,000 req/s.
- Aurora's Go runtime with goroutine-based concurrency delivers the lowest latency at every percentile while maintaining 100% success rate.
- Nginx-based gateways (Kong, APISIX) offer competitive throughput but higher tail latency due to worker process scheduling overhead.
- Node.js and Python gateways cannot sustain 5,000 req/s on consumer hardware — GC and GIL bottlenecks cap effective throughput below 2,500 req/s.
- Windows-specific scheduling overhead affects async Rust runtimes (tokio) more than Go's OS-thread-based goroutine scheduler.
- Configuration tuning (disabling usage tracking, request logging) provides a measurable but modest improvement — core architecture matters more.
Reproduce These Results
The benchmark suite ships with the Aurora repository. Run a full seven-gateway comparison on your own hardware using the provided scripts. All gateway configurations, startup scripts, and the mock server are included. Results are written as JSON comparison files with delta tables.