Deployment Guides / Configuration & Monitoring

Benchmark Setup

Run reproducible multi-gateway benchmarks comparing Aurora against Kong, APISIX, Bifrost, Helicone, Portkey, and LiteLLM.

Browse docs

Overview

The benchmark suite in scripts/ lets you compare Aurora against six other AI gateways on the same machine with identical request load. All gateways forward to a shared mock OpenAI-compatible backend so the test measures gateway overhead, not provider latency.

Two load-generator modes are available:

Rate-limited mode (default) — uses aurora-bench-cli to sustain a fixed requests-per-second target for a set duration
oha mode — concurrency-based load, no rate limiting, tests max throughput under connection pressure

Prerequisites

Requirement	Version	Check
Go	1.26+	`go version`
oha	any	`oha --version` (optional, for oha mode)
Docker	any	`docker --version` (for Kong, APISIX, Bifrost)
Node.js	20+	`node --version` (for Portkey)
Python	3.10+	`python --version` (for LiteLLM)
Git	any	`git --version`

Scripts

All benchmark scripts live under scripts/benchmarks/:

File	Purpose
`scripts/run-benchmark.ps1`	Entry point — builds Aurora, runs one or all gateways, generates comparison
`scripts/benchmarks/scenarios/multi-gateway.ps1`	Orchestrator — manages process lifecycle, preflight checks, warmup, profiling
`scripts/benchmarks/competitors/`	Per-gateway startup scripts (`aurora.ps1`, `kong.ps1`, `apisix.ps1`, `bifrost.ps1`, `helicone.ps1`, `portkey.ps1`, `litellm.ps1`)
`scripts/benchmarks/mock-server/main.go`	Lightweight OpenAI-compatible mock returning pre-encoded chat responses
`scripts/benchmarks/modules/BenchmarkComparison.psm1`	Delta calculation and summary tables
`scripts/benchmarks/modules/BenchmarkInfrastructure.psm1`	Process management, build helpers, health checks
`scripts/benchmarks/configs/`	Per-gateway config files (LiteLLM YAML, APISIX YAML, Kong YAML, Helicone YAML, Portkey config)

Quick Start

Run a smoke test (500 req/s for 30s) to verify everything works:

text

.\scripts\run-benchmark.ps1 -Mode smoke -NoWindow

This runs Aurora alone, handling 15,000 requests against the mock server. Results land in bench-results/.

Modes

Mode	Rate	Duration	Use Case
`smoke`	500 req/s	30s	Quick validation
`publish`	5,000 req/s	60s	Full throughput test (all gateways)
`custom`	`-Rate` / `-Duration`	as specified	Tuning and exploration

To run the full seven-gateway comparison:

text

.\scripts\run-benchmark.ps1 -Mode publish -NoWindow

This runs each gateway in sequence (Aurora, APISIX, Bifrost, Helicone, Kong, LiteLLM, Portkey), each at 5,000 req/s for 60s. The comparison JSON includes throughput, latency percentiles, and allocation metrics.

Running individual gateways

You can run individual gateways by passing the -Gateways parameter:

text

.\scripts\run-benchmark.ps1 -Mode custom -Rate 1000 -Duration 30 -Gateways @("aurora", "kong")

Running a Throughput Test (oha Mode)

oha sends requests at maximum speed without rate limiting — good for finding the real throughput ceiling:

text

.\scripts\run-benchmark.ps1 -Mode custom -Rate 10000 -Duration 10 -NoWindow -OhaMode -OhaRequests 100000 -OhaConcurrency @(100, 300, 500)

Concurrency Levels

Level	What It Tests
`100`	Light concurrency, low contention
`300`	Moderate concurrency, normal proxy load
`500`	High concurrency, saturating the gateway
`1000`	Extreme concurrency, scheduler contention
`2000`	Saturation — connection refusals expected

Protocol Notes

The benchmark applies HTTP/1.1 load to all gateways for fair comparison:

Aurora receives HTTP/1.1 requests but uses h2c (HTTP/2 cleartext) for its upstream connection to the mock server — this is its production default and provides better throughput via connection multiplexing
Kong, APISIX, Bifrost, Helicone, Portkey, LiteLLM — each uses its own upstream transport (HTTP/1.1 or HTTP/2 depending on the gateway)

Understanding the Output

Rate-Limited Mode Comparison

When using the official benchmark binary (default), a detailed JSON comparison and summary table is generated:

text

  Metric                     Aurora              Kong               Delta
  ---------------------------------------------------------------------------
  Throughput (req/s)          4,986              4,833              +153
  Success rate (%)            100.00              99.87             +0.13
  P50 latency (ms)              17.7               39.2            -21.5
  P99 latency (ms)              47.2               98.0            -50.8
  P999 latency (ms)             97.2              145.6            -48.4
  Allocs/op                     72.2               80.0             -7.8
  Bytes/op                   5,945              6,505             -560

Interpreting Gateway Differences

Signal	What It Means
Aurora near 100% target	Go's goroutine scheduler handles concurrent request processing efficiently
Kong/APISIX close behind	Nginx-based architectures with C core — low overhead for request forwarding
Bifrost ~60% target	Rust async overhead on Windows — tokio scheduler contention at 5K req/s
LiteLLM ~2% target	Python GIL + asyncio overhead — single-threaded event loop saturates at ~100 req/s
Portkey ~20% target	Node.js event loop + GC pauses limit sustained throughput
Zero Aurora errors	Graceful backpressure — no dropped requests under load

Interpreting pprof Profiles

Aurora captures CPU, heap, trace, and goroutine pprof profiles during the benchmark:

bench-results/<gateway>/<timestamp>.aurora.cpu.pprof
bench-results/<gateway>/<timestamp>.aurora.heap.pprof

Analyze them with:

text

go tool pprof -top -seconds 0 <path>.cpu.pprof
go tool pprof -top -alloc_space <path>.heap.pprof
go tool pprof -top -inuse_space <path>.heap.pprof

Common Hotspots

Profile Signal	Likely Cause
`runtime.cgocall` high	Windows syscall overhead from network I/O — expected on Windows
`encoding/json.(*decodeState).object`	JSON unmarshal — each request body is decoded for model validation
`net/http.(*http2clientStream).writeRequest`	h2c upstream multiplexing — framing overhead
`runtime.schedule` / `runtime.findRunnable`	Go scheduler contention at high concurrency

Typical Results

Rate-Limited (5,000 req/s)

Gateway	Req/s	P50 (ms)	P99 (ms)	Success
Aurora	4,986	17.7	47.2	100%
Kong	4,833	39.2	98.0	99.87%
APISIX	4,778	39.6	109.1	99.86%
Bifrost	2,970	75.4	285.8	99.79%
Helicone	2,398	102.0	217.3	99.68%
Portkey	978	243.0	480.6	99.40%
LiteLLM	117	1,565.8	24,175.5	56.74%

High Concurrency (oha, c=500)

Coming in a future benchmark run.

Troubleshooting

Symptom	Likely Fix
`gateway binary not found`	Download/built the gateway binary and place it in your `PATH`
`port already in use`	Previous run didn't clean up — kill any lingering gateway processes
`Aurora model registry not ready`	Check mock server is healthy and reachable
`oha not found`	Install via `scoop install oha` or `winget install oha`
Results differ across runs	Expected — benchmark variance from OS scheduling, background processes, and CPU throttling. Run 3 times and average.
LiteLLM port contention	LiteLLM blocks port for 60s+ after run — gateways tested after LiteLLM may show degraded latency

Reference

scripts/benchmarks/README.md — technical details on benchmark modules
/docs/about/benchmarks — summary of published Aurora benchmark data
cmd/bench/ — rate-limited benchmark CLI source
scripts/benchmarks/mock-server/main.go — mock OpenAI-compatible backend

← All docs

Deployment Guides / Configuration & Monitoring

Benchmark Setup

Run reproducible multi-gateway benchmarks comparing Aurora against Kong, APISIX, Bifrost, Helicone, Portkey, and LiteLLM.

Browse docs

Overview

Two load-generator modes are available:

Rate-limited mode (default) — uses aurora-bench-cli to sustain a fixed requests-per-second target for a set duration
oha mode — concurrency-based load, no rate limiting, tests max throughput under connection pressure

Prerequisites

Requirement	Version	Check
Go	1.26+	`go version`
oha	any	`oha --version` (optional, for oha mode)
Docker	any	`docker --version` (for Kong, APISIX, Bifrost)
Node.js	20+	`node --version` (for Portkey)
Python	3.10+	`python --version` (for LiteLLM)
Git	any	`git --version`

Scripts

All benchmark scripts live under scripts/benchmarks/:

File	Purpose
`scripts/run-benchmark.ps1`	Entry point — builds Aurora, runs one or all gateways, generates comparison
`scripts/benchmarks/scenarios/multi-gateway.ps1`	Orchestrator — manages process lifecycle, preflight checks, warmup, profiling
`scripts/benchmarks/competitors/`	Per-gateway startup scripts (`aurora.ps1`, `kong.ps1`, `apisix.ps1`, `bifrost.ps1`, `helicone.ps1`, `portkey.ps1`, `litellm.ps1`)
`scripts/benchmarks/mock-server/main.go`	Lightweight OpenAI-compatible mock returning pre-encoded chat responses
`scripts/benchmarks/modules/BenchmarkComparison.psm1`	Delta calculation and summary tables
`scripts/benchmarks/modules/BenchmarkInfrastructure.psm1`	Process management, build helpers, health checks
`scripts/benchmarks/configs/`	Per-gateway config files (LiteLLM YAML, APISIX YAML, Kong YAML, Helicone YAML, Portkey config)

Quick Start

Run a smoke test (500 req/s for 30s) to verify everything works:

text

.\scripts\run-benchmark.ps1 -Mode smoke -NoWindow

This runs Aurora alone, handling 15,000 requests against the mock server. Results land in bench-results/.

Modes

Mode	Rate	Duration	Use Case
`smoke`	500 req/s	30s	Quick validation
`publish`	5,000 req/s	60s	Full throughput test (all gateways)
`custom`	`-Rate` / `-Duration`	as specified	Tuning and exploration

To run the full seven-gateway comparison:

text

.\scripts\run-benchmark.ps1 -Mode publish -NoWindow

Running individual gateways

You can run individual gateways by passing the -Gateways parameter:

text

.\scripts\run-benchmark.ps1 -Mode custom -Rate 1000 -Duration 30 -Gateways @("aurora", "kong")

Running a Throughput Test (oha Mode)

oha sends requests at maximum speed without rate limiting — good for finding the real throughput ceiling:

text

.\scripts\run-benchmark.ps1 -Mode custom -Rate 10000 -Duration 10 -NoWindow -OhaMode -OhaRequests 100000 -OhaConcurrency @(100, 300, 500)

Concurrency Levels

Level	What It Tests
`100`	Light concurrency, low contention
`300`	Moderate concurrency, normal proxy load
`500`	High concurrency, saturating the gateway
`1000`	Extreme concurrency, scheduler contention
`2000`	Saturation — connection refusals expected

Protocol Notes

The benchmark applies HTTP/1.1 load to all gateways for fair comparison:

Aurora receives HTTP/1.1 requests but uses h2c (HTTP/2 cleartext) for its upstream connection to the mock server — this is its production default and provides better throughput via connection multiplexing
Kong, APISIX, Bifrost, Helicone, Portkey, LiteLLM — each uses its own upstream transport (HTTP/1.1 or HTTP/2 depending on the gateway)

Understanding the Output

Rate-Limited Mode Comparison

When using the official benchmark binary (default), a detailed JSON comparison and summary table is generated:

text

  Metric                     Aurora              Kong               Delta
  ---------------------------------------------------------------------------
  Throughput (req/s)          4,986              4,833              +153
  Success rate (%)            100.00              99.87             +0.13
  P50 latency (ms)              17.7               39.2            -21.5
  P99 latency (ms)              47.2               98.0            -50.8
  P999 latency (ms)             97.2              145.6            -48.4
  Allocs/op                     72.2               80.0             -7.8
  Bytes/op                   5,945              6,505             -560

Interpreting Gateway Differences

Signal	What It Means
Aurora near 100% target	Go's goroutine scheduler handles concurrent request processing efficiently
Kong/APISIX close behind	Nginx-based architectures with C core — low overhead for request forwarding
Bifrost ~60% target	Rust async overhead on Windows — tokio scheduler contention at 5K req/s
LiteLLM ~2% target	Python GIL + asyncio overhead — single-threaded event loop saturates at ~100 req/s
Portkey ~20% target	Node.js event loop + GC pauses limit sustained throughput
Zero Aurora errors	Graceful backpressure — no dropped requests under load

Interpreting pprof Profiles

Aurora captures CPU, heap, trace, and goroutine pprof profiles during the benchmark:

bench-results/<gateway>/<timestamp>.aurora.cpu.pprof
bench-results/<gateway>/<timestamp>.aurora.heap.pprof

Analyze them with:

text

go tool pprof -top -seconds 0 <path>.cpu.pprof
go tool pprof -top -alloc_space <path>.heap.pprof
go tool pprof -top -inuse_space <path>.heap.pprof

Common Hotspots

Profile Signal	Likely Cause
`runtime.cgocall` high	Windows syscall overhead from network I/O — expected on Windows
`encoding/json.(*decodeState).object`	JSON unmarshal — each request body is decoded for model validation
`net/http.(*http2clientStream).writeRequest`	h2c upstream multiplexing — framing overhead
`runtime.schedule` / `runtime.findRunnable`	Go scheduler contention at high concurrency

Typical Results

Rate-Limited (5,000 req/s)

Gateway	Req/s	P50 (ms)	P99 (ms)	Success
Aurora	4,986	17.7	47.2	100%
Kong	4,833	39.2	98.0	99.87%
APISIX	4,778	39.6	109.1	99.86%
Bifrost	2,970	75.4	285.8	99.79%
Helicone	2,398	102.0	217.3	99.68%
Portkey	978	243.0	480.6	99.40%
LiteLLM	117	1,565.8	24,175.5	56.74%

High Concurrency (oha, c=500)

Coming in a future benchmark run.

Troubleshooting

Symptom	Likely Fix
`gateway binary not found`	Download/built the gateway binary and place it in your `PATH`
`port already in use`	Previous run didn't clean up — kill any lingering gateway processes
`Aurora model registry not ready`	Check mock server is healthy and reachable
`oha not found`	Install via `scoop install oha` or `winget install oha`
Results differ across runs	Expected — benchmark variance from OS scheduling, background processes, and CPU throttling. Run 3 times and average.
LiteLLM port contention	LiteLLM blocks port for 60s+ after run — gateways tested after LiteLLM may show degraded latency

Reference

scripts/benchmarks/README.md — technical details on benchmark modules
/docs/about/benchmarks — summary of published Aurora benchmark data
cmd/bench/ — rate-limited benchmark CLI source
scripts/benchmarks/mock-server/main.go — mock OpenAI-compatible backend