Benchmarking AI gateways is fundamentally different from benchmarking web servers or databases. The request path involves protocol parsing, model routing, provider-specific transformations, streaming, and observability hooks — all of which add overhead that must be measured precisely.
This guide covers the methodology we use at Aurora to produce reliable, reproducible gateway benchmarks. Apply the same approach to evaluate any gateway in your own infrastructure. All examples reference our seven-gateway comparison, which tested Aurora, Kong, APISIX, Bifrost, Helicone, Portkey, and LiteLLM under identical conditions.
Why Rate-Limited Testing Matters
The most common benchmarking mistake is using concurrency-based load generators (like oha or wrk) that blast requests as fast as possible. This measures the gateway's saturation point, but it does not measure performance under controlled load — which is what production traffic looks like.
Rate-limited testing sends requests at a fixed target rate (e.g., 5,000 req/s) and measures how well the gateway keeps up. This reveals:
- Whether the gateway can sustain the advertised throughput.
- How latency degrades as the gateway approaches its capacity limit.
- At what point the gateway starts dropping or timing out requests.
A gateway that achieves 10,000 req/s in a saturation test may fail at 3,000 req/s under rate-limited load if its request scheduling is inefficient. In our comparison, Aurora maintained 4,986 req/s at a 5,000 target — 99.7% efficiency — while LiteLLM achieved only 117 req/s (2.3% efficiency) at the same target rate.
Essential Metrics
Throughput (Requests Per Second)
The number of successfully completed requests per second. Compare this against the target rate to calculate efficiency. Throughput alone does not tell the full story — it must be paired with latency percentiles and success rate.
Gateway Target Achieved Efficiency Aurora 5,000/s 4,986/s 99.7% Kong 5,000/s 4,833/s 96.7% APISIX 5,000/s 4,778/s 95.6% Bifrost 5,000/s 2,970/s 59.4% Helicone 5,000/s 2,398/s 48.0% Portkey 5,000/s 978/s 19.6% LiteLLM 5,000/s 117/s 2.3%
Latency Percentiles (P50, P90, P95, P99, P999)
Median latency (P50) represents typical performance. P99 and P999 represent tail latency — the worst-case experience for 1 in 100 and 1 in 1,000 requests respectively. Most gateway users notice P99 far more than average latency.
Aurora's P99 of 47.2 ms means 99% of requests complete in under 47.2 ms. LiteLLM's P99 of 24.2 seconds means 1 in 100 requests waits over 24 seconds — a catastrophic user experience. Always report at least P50, P99, and P999.
Success Rate
The percentage of requests that complete with a 200-level HTTP status. Errors under load are typically timeouts or connection resets, both of which indicate the gateway cannot keep up with the request rate. Aurora achieved 100% success rate across 299,182 requests — no other gateway in the comparison matched this.
Allocations Per Operation
Memory allocation efficiency directly impacts throughput. Fewer allocations mean less GC pressure, which translates to lower and more stable latency under load. Measure allocations per request (allocs/op) and bytes per request (bytes/op).
Aurora and APISIX tied at 72 allocs/op, while Helicone required 91 allocs/op and LiteLLM required 103 allocs/op. Over millions of requests, a difference of 20-30 allocs/op translates to significant GC overhead.
Testing Environment Controls
To produce reproducible results, control the following variables:
- Hardware: Use the same machine for all gateways in a comparison. Document CPU model, core count, RAM, and OS. Our comparison used an Intel Core i5-10300H (4 cores, 8 threads) with 8 GB RAM on Windows 10.
- Mock upstream: All gateways must forward to the same mock server. Never test against a real LLM provider — network latency and API rate limits will distort results. Our mock server was verified to handle 10,000+ req/s independently.
- Request payload: Use identical payload sizes and shapes for every gateway. Both the request and response size affect throughput. Our test used identical gpt-4o-mini chat completion requests throughout.
- Warmup: Send a warmup burst of requests before measurement begins. Our warmup used 25 requests plus 256 prewarm payloads to initialize connection pools and caches.
- Run duration: Run each test for at least 60 seconds. Short runs miss GC-induced latency spikes. Our comparison used 60-second runs at each rate.
- Test order: Test order matters — gateways that block ports after shutdown (notably LiteLLM) degrade subsequent runs. Test the fastest gateways first, or reset network state between tests.
Run-to-Run Variance
No single benchmark run is perfectly reliable. Operating system scheduling, background processes, CPU thermal throttling, and memory pressure all introduce variance. In our comparisons, Aurora's throughput varied by less than 2% between runs at the same target rate, while Python-based LiteLLM showed variance exceeding 20%.
A dedicated three-gateway comparison (Aurora, Kong, Bifrost) run earlier in the same session showed slightly different absolute numbers — Kong achieved 3,757 req/s instead of 4,833 req/s in the full comparison, likely due to LiteLLM port contention in the longer run. This highlights the importance of running dedicated comparisons with controlled test ordering.
Common Pitfalls
Testing Against Real Providers
Gateway benchmarks must use a mock upstream. Real LLM providers introduce uncontrollable variables: network jitter, rate limits, model loading latency, and multi-tenancy noise. A gateway may appear 2x slower simply because the provider responded slower during its test window.
Ignoring Warmup Effects
Go runtimes need time for goroutine scheduler warmup. Python runtimes need interpreter warmup. Node.js runtimes need V8 optimization. Without warmup, early-second measurements will be misleadingly slow. Our warmup phase discarded the first 25 requests plus 256 payloads before recording.
Single-Run Confidence
Always run each benchmark at least three times and report the median. Operating system scheduling, background processes, and CPU thermal throttling introduce variance that a single run cannot capture. In practice, we found that Aurora results were consistent within 1-2% across runs, while interpreted gateways showed 5-15% variance.
Conflating Concurrency With Throughput
High concurrency does not mean high throughput. A gateway handling 500 concurrent connections may achieve 2,000 req/s while another handling 100 connections achieves 5,000 req/s. Always report throughput (req/s) alongside concurrency. Rate-limited testing avoids this confusion entirely by fixing the request rate.
Port Contamination
When testing multiple gateways sequentially, ensure each gateway fully releases its port before the next starts. On Windows, TIME_WAIT socket states can keep ports bound for 60+ seconds. We observed LiteLLM blocking port 8082 for over 60 seconds after shutdown, degrading latency measurements for gateways tested immediately afterward.
Benchmark Configuration Template
Below is the actual configuration used in our seven-gateway comparison. Each gateway received the same payload targeting the gpt-4o-mini model through the /v1/chat/completions endpoint:
# Test parameters RATE=5000 DURATION=60 MODEL=gpt-4o-mini ENDPOINT=/v1/chat/completions BIG_PAYLOAD=false # Warmup PREWARM_COUNT=25 PREWARM_PAYLOADS=256 # Mock upstream (port varies per gateway) MOCK_PORT=8081 MOCK_MODEL=gpt-4o-mini # Gateway ports AURORA_PORT=8081 KONG_PORT=8000 APISIX_PORT=9080 BIFROST_PORT=8080 HELICONE_PORT=8585 PORTKEY_PORT=8787 LITELLM_PORT=8082 # Timing TIMER=QPC # Query Performance Counter (Windows) REPORT_INTERVAL=100ms # Metric reporting interval
Interpreting Results
When comparing gateways, look at the full picture rather than a single metric:
- A gateway with 100% success rate and slightly higher latency is preferable to one with lower latency but 0.5% error rate. Aurora's 100% success rate at 4,986 req/s is more valuable than a hypothetical 5,200 req/s with 1% errors.
- P99 latency matters more than P50 for user-facing applications. Aurora's 17.7 ms P50 with 47.2 ms P99 means consistent performance across all users. LiteLLM's 1.6 s P50 with 24.2 s P99 means most users are waiting seconds and 1% wait over 24 seconds.
- Memory allocation efficiency (allocs/op) predicts how the gateway will behave under sustained load over hours, not just 60-second runs. A difference of 20 allocs/op between Aurora (72) and Bifrost (92) translates to 100,000 extra allocations per second at 5,000 req/s — enough to trigger meaningful GC pressure.
- Throughput efficiency (achieved req/s vs target req/s) reveals how close the gateway operates to its architectural limits. Above 95% efficiency indicates the gateway's architecture is well-matched to the load. Below 50% suggests fundamental bottlenecks (GIL, GC, scheduler contention).
Key Takeaways
- Always use rate-limited testing, not concurrency-based saturation testing, for reproducible gateway benchmarks.
- Measure latency percentiles (P50, P99, P999) — average latency hides tail latency problems.
- Use a mock upstream to eliminate provider network variance — verify the mock can handle 2x your target rate.
- Run each test at least three times and report the median — expect 1-2% variance for compiled gateways and 5-15% for interpreted runtimes.
- Document hardware specs (CPU model, core count, RAM, OS), test duration, warmup, and payload sizes for reproducibility.
- Allocations per operation (allocs/op) predicts long-run stability under sustained load — track it alongside throughput and latency.
- Test order matters: run the fastest gateways first to avoid port contamination from slower gateways that block ports after shutdown.