Reducing Inference Cost by 25–60% with Model Routing + Token Budgets (Quality Held Steady)

Inference costs were spiraling. Context bloat, always using the largest model, retries, and no caching drove the bill up. We implemented LLM cost optimization strategies (see Reduce OpenAI Bill Without Hurting Quality for the audit framework): model routing, context compression, tool-calling guardrails, and caching. Cost per task dropped 25–60%; quality and p95 stayed stable.

Executive summary

The client had an LLM system with high inference costs. Every request used the largest model; context was bloated; retries and caching were absent. We implemented model routing (small/large), context compression, tool-calling guardrails, and caching. Cost per task dropped 25–60%; quality and p95 latency held steady.

This is the pattern behind LLM cost too high: teams blame model pricing when the real issue is routing discipline and context control. Use Cost Optimization to frame the full operating model before you jump into vendor swaps.

Baseline (before)

Before optimization:

• Cost per task: High—inference was the main cost driver
• Tokens per request: Bloated—full context sent every time
• Model usage: Always used largest model—regardless of task complexity
• Caching: None—repeated/similar queries paid full cost

Diagnosis

We traced cost drivers:

• Context bloat: Full documents sent when summaries would suffice
• Always-LLM-large: Simple tasks routed to expensive model
• Retries: Failed requests retried without cost awareness
• No caching: Repeated/similar queries paid full inference cost

The fix

We implemented a structured cost optimization plan:

• Model routing: Route simple tasks to small model; complex tasks to large model
• Context compression: Summarize or trim context before sending to LLM
• Tool-calling guardrails: Limit tool calls to reduce token usage
• Caching: Cache retrieval and LLM responses for repeated/similar queries
• Prompt trimming: Remove redundant instructions and examples

Metrics

Why this worked

Model routing sent simple tasks to cheaper models. Context compression reduced token count. Caching avoided redundant inference. Tool-calling guardrails prevented runaway token usage. Quality was preserved because we didn't sacrifice accuracy—we optimized where it didn't matter.

What we shipped

Artifacts delivered to the client:

• Token spend decomposition dashboard (by model, surface, cohort)
• Model routing policy (intent classifier + escalation rules)
• Context compression pipeline (summarization + trimming)
• Response cache layer with TTL and invalidation
• Tool-call guardrails (max steps, retry budget)
• Cost per task baseline + post-fix metrics
• Eval harness for quality regression checks
• Before/after benchmark report (exec-ready)

Example artifact: eval output table

Dashboard screenshot (redacted): cost per task, quality score, P95 trends.

---

Next steps

If your LLM system has high inference costs, an AI system audit can baseline cost per task and deliver a prioritized optimization plan. We help teams build model routing, token budgets, and caching—with before/after validation. If you need stakeholder buy-in first, start with the cost pain page.