Provider Prompt Caching

What it actually does

When you send a chat request with a system prompt, a long first user message, and a tool list, the upstream provider may have to re-encode those tokens on every call. Most providers offer a way to mark parts of the prompt as cacheable, charge you less for the cached portion, and reuse the encoding across requests.

Aurora's provider prompt caching inserts those cache_control markers for you when the upstream supports them. This is separate from Aurora's own response cache — the response cache avoids calling the provider at all, while prompt caching reduces the cost of the call that does happen. See Cache.

Per-provider support

What the gateway inserts

In default mode, Aurora's applyPromptCache runs per request and:

1. Estimates the request size as total_chars / 4 and skips the inserts if the request is below 1024 estimated tokens. The threshold is a code constant, not a config knob.
2. For Anthropic, inserts cache_control: {type: "ephemeral"} on:
   • The first system message (if any)
   • The first user message (if any)
   • The request-level tools array (if any)
3. For OpenAI-compatible providers, inserts the same cache_control markers via the request ExtraFields round-trip — upstream providers that recognize them will cache; upstream providers that ignore them will just see the field and continue.
4. For Gemini, attempts the same and then logs a note that explicit cachedContent is the only way to get a real Gemini cache hit; use passthrough for that.
5. Returns immediately for Groq and DeepSeek.

How the cost shows up

After a successful cached call, the response usage block carries the saved tokens:

Anthropic (response usage):

{
  "input_tokens": 1500,
  "cache_creation_input_tokens": 1500,
  "cache_read_input_tokens": 0,
  "output_tokens": 200
}

On the second and later calls, cache_read_input_tokens increases and you pay the lower cached-input rate.

OpenAI (response usage):

{
  "prompt_tokens": 1500,
  "completion_tokens": 200,
  "total_tokens": 1700,
  "prompt_tokens_details": {
    "cached_tokens": 1500
  }
}

cached_tokens is what the upstream charged at the cache rate; the rest was billed at full price.

Cached token pricing

Aurora's pricing model supports a separate cached_input_per_mtok per model so dashboard cost reports reflect the discounted rate:

providers:
  - name: anthropic-prod
    type: anthropic
    api_key: "${ANTHROPIC_API_KEY}"
    models:
      - id: claude-sonnet-4-5
        pricing:
          input_per_mtok: 3.0
          cached_input_per_mtok: 0.30
          output_per_mtok: 15.0

If cached_input_per_mtok is not set, Aurora treats cached tokens as zero-cost in the dashboard. This is rarely what you want — set it for any model you actually cache against.

Disabling it per request

Set the request-level cache-control to disable Aurora's auto-inserts for a single call:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AURORA_MASTER_KEY" \
  -d '{
    "model": "anthropic/claude-sonnet-4-5",
    "messages": [{"role": "user", "content": "Hello"}],
    "cache_control": {"type": "off"}
  }'

With cache_control.type: "off", Aurora still forwards the field to the upstream so the upstream can decide what to do, but the gateway does not insert its own auto-breakpoints.

How this interacts with the gateway response cache

The two caches compose:

1. Response cache — checked first on a non-streaming request. On a hit, no provider call is made and no prompt cache events occur.
2. Provider prompt cache — applies only when the request actually reaches the provider. Reduces the cost of that call by reusing encoded prompt segments.

You typically want both enabled for a high-traffic RAG or tool-using workload: the response cache takes care of exact replays, the prompt cache takes care of the calls that do go through but share a long system prompt or tool list.

When it does not help

• First call — there is nothing to read from the cache yet. The first call may be slightly more expensive (cache_creation_input_tokens) and the savings start on call two.
• Short prompts — requests below 1024 estimated tokens skip the inserts because the breakpoint overhead would be larger than the savings.
• Streaming-only responses — most providers still credit cache hits on streaming responses, but check the provider's billing page if this matters to you.
• No system prompt, no tools, short user message — there is nothing stable to cache. Aurora's inserts are no-ops in that case.

Gotchas

• The 1024-token threshold is hard-coded. If you need a different threshold, edit internal/promptcache/prompt_cache.go and rebuild. There is no env var.
• Tools cache only applies when there is at least one tool. No tool array, no tools breakpoint.
• Cache reads are not free on every provider. Some providers charge a small read fee, others discount aggressively. Check the provider's pricing page, not Aurora's.
• Caching is per-prefix. Move your stable system prompt to the start of the messages array, your tool list stays attached to the request, and your per-request content stays at the end. Reordering breaks the cache.
• Bumping the system prompt invalidates downstream caches. If you A/B test system prompts, the prompt cache for the old prompt is dead weight for as long as you keep testing.
• Manual mode is request-driven. When the gateway is in manual mode, it only inserts breakpoints for requests that carry an explicit cache_control: {type: "ephemeral"} field. In auto mode it always inserts them. The mode itself is not a config knob in this build.

Related

• Cache — gateway-level response cache
• Pricing — cached_input_per_mtok overrides
• Passthrough API — for Gemini's native cachedContent and other features that need a raw upstream call