Token Saver

Token Saver is Aurora's policy-driven cost control. It can (a) compress eligible prompt and tool-output content before sending to the provider, and (b) append a concise-response style instruction so the provider returns shorter answers. Both are opt-in per model, provider, and endpoint.

When to turn it on

Turn Token Saver on when:

• Your app makes high-volume calls to expensive models (Claude Opus, o3, etc.) and you want a per-call spend reduction.
• You pipe tool output into the prompt and the tool output is often larger than the model's context budget.
• You want consistent concise-style answers without hardcoding a system prompt in your app.

Leave it off when:

• You are doing creative generation where the model benefits from long answers.
• Latency-sensitive traffic where any extra processing time matters.
• You already control verbosity via system prompts and want deterministic behavior.

Enable

TOKEN_SAVER_ENABLED=true

By default, Token Saver applies compression only to /v1/chat/completions. Configure the rest below.

Output style profiles

Token Saver can request a specific answer style from the model by appending a system instruction. Profiles:

Enable output style:

TOKEN_SAVER_OUTPUT_ENABLED=true
TOKEN_SAVER_OUTPUT_PROFILE=concise

The profile is applied as a system-instruction decoration, not a rewrite — your existing system prompt is preserved.

Scoping

Apply Token Saver only to specific models or providers:

TOKEN_SAVER_MODELS_INCLUDE=gpt-4o,claude-sonnet-4-5
TOKEN_SAVER_MODELS_EXCLUDE=o3-mini
TOKEN_SAVER_PROVIDERS_INCLUDE=openai-primary,anthropic-prod
TOKEN_SAVER_PROVIDERS_EXCLUDE=groq

The include list is "only these"; the exclude list is "anything but these". Exclude wins when both are set.

Endpoints

By default, Token Saver runs on /v1/chat/completions. Add more:

TOKEN_SAVER_ENDPOINTS=chat_completions,responses

Valid values: chat_completions, responses.

Streaming

TOKEN_SAVER_APPLY_STREAMING=true

When on, Token Saver compresses the prompt and applies the output style to streaming responses too. The compression runs on the assembled final answer, so streaming remains true SSE — there is no per-delta modification.

Error behavior

TOKEN_SAVER_ON_ERROR=allow

• allow (default) — if Token Saver fails, the request still proceeds with the unmodified prompt.
• block — if Token Saver fails, the request returns 502. Use this when you want strict accounting and never want a request to go upstream uncompressed.

Audit

TOKEN_SAVER_AUDIT_ENABLED=true
TOKEN_SAVER_EMIT_HEADERS=true

When emit_headers is on, every response includes:

• X-Aurora-Token-Saver: applied — token saver modified the request
• X-Aurora-Token-Saver: skipped — token saver was eligible but did not modify (e.g. below the min-tokens threshold, or excluded by model/provider include/exclude)
• X-Aurora-Token-Saver-Skip: <reason> — set when the value is skipped; the reason explains why
• X-Aurora-Token-Saver-Profile: <profile> — set when an output profile (currently caveman) was applied

The header is omitted entirely when token saver is disabled, or when emit_headers is off.

When audit is on, the action is also recorded in the audit log so you can confirm what was changed per request.

Sample request and response

Request:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AURORA_MASTER_KEY" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Explain quantum entanglement in one paragraph."}]
  }'

Response headers (with TOKEN_SAVER_OUTPUT_ENABLED=true, TOKEN_SAVER_OUTPUT_PROFILE=concise):

HTTP/1.1 200 OK
X-Aurora-Token-Saver: applied
X-Aurora-Token-Saver-Profile: concise

The body is a normal chat completion; Token Saver did not modify the user prompt, only added a system-instruction decoration. Set TOKEN_SAVER_OUTPUT_ENABLED=false to disable the decoration but keep prompt compression.

Gotchas

• First token latency — Token Saver adds a small amount of processing before the upstream call. On very fast providers (Groq), this can be measurable. Disable with TOKEN_SAVER_APPLY_STREAMING=false and TOKEN_SAVER_ENABLED=false for latency-critical paths.
• Provider-specific behavior — output style is a system-instruction decoration. The provider's compliance with the style varies; measure with the response length metric in the dashboard before treating ultra as a hard cap.
• Workflows win — if a workflow for a (provider, model) turns Token Saver off, Token Saver is off for that traffic regardless of the env var.
• Caching interaction — Token Saver modifies the prompt, so the cache key changes. If you turn Token Saver on after building a cache, expect a cold start.

Related

• Observability — measure spend before and after enabling
• Cache — different problem: avoid the call entirely
• Pricing — override per-model prices