Provider Pools.
Intelligent multi-provider routing with load balancing, health-aware failover, and forced routing controls.
What Are Provider Pools
A provider pool is a group of provider configurations that serve the same model capability. Instead of hard-coding a single provider, you define a pool of candidates and Aurora automatically selects the best member for each request based on the configured strategy, current health, and load. Pools enable zero-downtime failover, cost optimisation, and performance balancing across providers.
Load Balancing Strategies
Each pool is configured with a strategy that determines how incoming requests are distributed among pool members:
round_robin
Requests are distributed sequentially across members in order. Simple and predictable, but does not account for member capacity.
least_loaded
Routes to the member with the fewest active connections. Best for heterogeneous members where some may be slower than others.
weighted
Each member gets a weight value. Members with higher weights receive proportionally more traffic. Useful for cost-preference routing.
random
Randomly selects a member for each request. Provides good distribution under high concurrency with minimal overhead.
adaptive
Monitors real-time latency and error rates per member, dynamically shifting traffic toward the best-performing members. Self-tuning over time.
Health-Aware Routing
Every pool member is continuously monitored. When a member's circuit breaker opens (due to excessive errors or timeouts), it is automatically skipped during member selection. Once the cooldown period expires and health checks pass, the member is re-added to rotation. This ensures that failing providers never degrade gateway reliability.
Failover Within Pool
When a pool member returns a transient error (rate limit, timeout, 5xx), Aurora automatically retries the request with the next available member in the pool. The failover happens within the same request lifecycle — no client-side retry logic needed. You can configure the maximum number of failover attempts and which error codes trigger a failover.
Direct Provider Override
Clients can bypass pool selection by sending theX-Provider-Key-Idheader with a specific provider key ID. When present, the gateway routes directly to that provider, ignoring the pool's load balancing strategy. This is useful for testing, debugging, or when a specific provider is required for compliance.
Auth Key Provider Pool ID
API keys can be assigned a provider_pool_id at creation time. When a request uses such a key, the gateway forces routing through the specified pool, overriding any model-based pool resolution. This enables per-customer provider routing without changing application code.