Enterprise pain page

LLM cost too high? Cut inference spend (hosted APIs + self-hosted)

The problem is rarely just “too many tokens.” Bills spike when weak workflows create extra turns, retries, tool loops, and more human review. We decompose spend, tie it to successful outcomes, and fix the real drivers first.

OpenAIAnthropic ClaudeGoogle GeminiAWS BedrockAzure OpenAISelf-hosted (vLLM / TGI / SGLang)On-prem GPU

Start with AI Production Audit Get the rollout checklist Talk to a technical lead

Symptoms: what teams observe

Monthly spend rising without clear attribution. Bills spike after prompt or model changes. Cost per ticket or per user is unacceptable. Leadership asks for ROI and you can't explain where the money goes.

OpenAI / Claude / Gemini / Bedrock spend climbing month-over-month

Self-hosted GPU burn with unclear utilization / ROI

Cost per conversation or per ticket too high

No breakdown: which model, which surface, which cohort

Retries and failures still costing

Tool-calling agents 2–5x more expensive than expected

Context bloat: full docs sent when summaries would do

Why it happens: failure modes

Without instrumentation, you can't separate input-token waste from output-token waste, retries from successes, or tool loops from single-shot calls. We trace spend into buckets and identify the dominant drivers.

Context bloat

Full documents, long history, over-fetched RAG. Input tokens dominate.

Always-LLM-large

Simple tasks routed to expensive models. No intent-based routing.

Retries & failures

Failed requests still cost. Timeouts, rate limits trigger retries.

Tool loops

Each tool invocation = extra round-trip. Agentic workflows multiply cost.

No caching

Repeated or near-identical queries pay full inference every time.

Infra waste (self-hosted)

Low GPU utilization, poor batching, low cache hit-rate, mis-sized instances, noisy neighbors.

How we diagnose

We need request-level attribution: input vs output tokens, system vs history vs RAG vs tool, retry count, tool calls per request—and (for self-hosted) GPU utilization and throughput metrics. If you don't have this yet, we implement minimum logging as part of the audit. For a concrete walkthrough, see our token spend decomposition guide (OpenAI example).

Required for diagnosis

• Token breakdown: input vs output, by source (system, history, RAG, tool)
• Retry count and status per request
• Tool calls count and agent steps
• Model and surface (or cohort) labels
• Self-hosted (if applicable): GPU utilization, throughput, queueing, batch size

Fix order (step-by-step)

We prioritize by impact. Usually: (1) instrument + baseline, (2) model routing (small vs large by intent), (3) context trimming and RAG pruning, (4) caching for repeated/similar queries, (5) tool-call guardrails and retry policies.

1. Instrument + baseline: token decomposition, cost per successful task

2. Model routing: route simple tasks to small model; escalate only when needed

3. Context trimming: summarize, truncate, right-size RAG top-k

4. Caching: response cache, prompt cache where latency allows

5. Tool guardrails: limit tool calls, retry budgets, stop conditions

What "good" looks like

Before/after metrics. Cost per successful task down 25–60%. Quality held or improved. P95 latency stable. Exec-ready narrative: "We cut cost X% with no quality drop."

Example: before/after

Metric	Before	After
Cost per task	High	25–60% lower
Tokens per request	Bloated	Reduced
Quality score	Baseline	Held steady

Ready to reduce LLM cost?

Start with an AI Production Audit to baseline cost and get a prioritized fix plan. Or jump to an Optimization Sprint if you already have a baseline.

Request AI Production Audit Get the rollout checklist Cost Optimization Hub

FAQ

Why is my LLM cost so high?

Common drivers: context bloat (full docs when summaries suffice), always-LLM-large (simple tasks routed to expensive models), retries (failed requests still cost), no caching (repeated queries pay full price), tool loops (multi-step agents multiply cost). We trace spend by bucket and fix the dominant drivers.

Does this apply to self-hosted / open-source models too?

Yes. For self-hosted models, “cost” often hides in GPU utilization and throughput: oversized prompts, low cache hit-rate, poor batching, inefficient tool loops, and retry storms—plus infra waste (underutilized GPUs, mis-sized instances, noisy neighbors). We treat hosted and self-hosted the same way: cost per successful task, traced to the dominant drivers.

How do you reduce LLM cost without losing quality?

We use model routing (small vs large by intent), token/context optimization, and caching. Every change is validated with before/after benchmarks. Cost per successful task drops; quality holds or improves.

What is an AI Production Audit for cost?

A 1–2 week baseline: we instrument your system, decompose token spend by bucket (system, history, RAG, tool, retries), identify the top cost drivers, and deliver a prioritized fix roadmap. Often the first step before an Optimization Sprint.

How fast can we see cost reduction?

Typical Optimization Sprint: 4–6 weeks. We ship PRs for routing, caching, context trimming, and tool guardrails. Before/after metrics validate impact. Many clients see 25–60% cost reduction with quality held steady.

Recommended next

Part of Cost Optimization Hub.

Request AI Production Audit