GenAI Audit vs AI System Audit: Scope, Artifacts, and Proof

GenAI Audit vs AI System Audit: scope, artifacts, and how to prove before/after impact. Choose the right audit type for your production stage.

When teams say they need a "GenAI audit," they can mean anything from a quick prompt review to a full production teardown. The real question: Are we auditing "AI" in the abstract—or the actual system that ships answers in production?

This post clarifies the difference between a GenAI Audit and an AI System Audit, what artifacts you should expect, and how to prove impact with before/after evidence—not opinions.

1) The short distinction

GenAI Audit (often)

A GenAI audit typically focuses on the model-facing layer: prompts (system + user), model choice, safety prompts/guardrails (basic), example conversations. It's useful early, but it can miss what actually breaks production.

AI System Audit (what you need in production)

An AI System Audit treats your product as a pipeline: inputs → retrieval → context construction → generation → tools/actions → output validation → monitoring

It covers the whole chain and answers: Why do we get wrong answers? Where does latency spike? Why is cost exploding? What regresses when we ship changes?

2) Why this matters: most "AI problems" are system problems

In production, you don't ship a model—you ship a system. A great prompt can't fix:

• low-recall retrieval
• stale knowledge
• tool timeouts + retry storms
• missing eval gates
• lack of observability (you can't see failure modes)
• routing policies that send easy queries to expensive models

If your pain is quality/cost/latency/reliability, you want an AI System Audit.

3) The scope matrix (what each audit should include)

1) Quality & correctness

GenAI Audit usually checks: prompt clarity, instruction hierarchy, output formatting consistency, basic safety/refusal behavior.

AI System Audit adds: failure taxonomy (top failure modes with real examples), cohort breakdown (intent, language, tenant, doc type), "retrieval vs generation vs tool" classification, groundedness / citation validity sampling.

2) RAG & knowledge (if you use RAG)

GenAI Audit might: glance at chunking and prompts, suggest "try reranking" or "increase k."

AI System Audit actually measures: retrieval recall proxies, ranking quality, context construction issues (wrong chunks stitched, contradictions), freshness gaps, cost/latency tradeoffs (k, rerank, query rewrite).

3) Cost

GenAI Audit often: suggests shorter prompts, suggests a cheaper model.

AI System Audit produces: token spend decomposition (input/output, retries, tool loops, RAG costs), cost per successful task (not cost per request), cost drivers by cohort/intent, a prioritized ROI roadmap.

4) Latency & serving

GenAI Audit: "the model is slow," "use streaming," "use smaller model."

AI System Audit: stage-level latency breakdown (retrieval, rerank, tools, generation), TTFT vs total latency analysis, queueing symptoms at peak hours, timeout/retry policy review (often the hidden killer), bottleneck chain.

5) Reliability & regressions

GenAI Audit might: suggest better prompt structure, propose "more tests."

AI System Audit: builds or refactors an evaluation harness plan, identifies regression risks across cohorts, defines release gates (CI regression tests), sets monitoring KPIs + drift detection.

6) Security & compliance (especially for enterprise)

GenAI Audit: "don't log sensitive data," basic prompt injection notes.

AI System Audit: threat model for your actual agent/RAG/tool surface, logging policy, PII redaction & data minimization guidance, tool allowlists, output validation, injection defenses, vendor risk questions (data usage/training/retention).

4) The artifact checklist (what you should expect to receive)

If you pay for an audit, you should come away with artifacts you can use immediately.

Minimum deliverables for a real AI System Audit:

• Baseline scorecard: quality, groundedness, cost/task, p95 latency (segmented by cohorts)
• Failure taxonomy: top 10–20 failure modes with examples
• Root-cause map: retrieval vs generation vs tools vs policies
• Cost teardown: token spend + retry/tool drivers + "silent spenders"
• Latency teardown: stage-level breakdown + queueing/timeout issues
• Roadmap: quick wins + deeper fixes + expected ROI/risk
• Eval plan: what to gate in CI, what to monitor daily
• Instrument plan: minimum logging/tracing schema to debug reliably
• Security notes (at least baseline): injection/tool misuse/PII posture

If an audit doesn't produce usable artifacts, it's not an audit—it's a review.

Minimum logging schema (so you can debug "retrieval vs generation vs tools")

If you can't reproduce failures with traces, every bug becomes a prompt patch. This is the minimum trace shape we recommend (store redacted/hashes where needed).

{
  "trace_id": "01J0X8Y9Q3KJ3V7R7H1M2N3P4Q",
  "timestamp": "2026-02-15T10:32:11.402Z",
  "env": "prod",
  "app_version": "2026.02.15",
  "tenant_id": "acme-enterprise",
  "user_id_hash": "sha256:7c9e...b13a",
  "session_id": "s_9f2c1a",
  "request": {
    "channel": "web",
    "locale": "vi-VN",
    "intent": "policy_qa",
    "query": "Do we log customer PII in chat history?",
    "attachments_count": 0
  },
  "routing": {
    "policy": "fast_or_smart_v2",
    "selected_model": "gpt-4.1-mini",
    "fallback_model": "gpt-4.1",
    "reason_codes": ["enterprise", "needs_citations"]
  },
  "retrieval": {
    "enabled": true,
    "corpus": "kb_prod",
    "query_rewrite": {
      "enabled": true,
      "rewrite": "customer PII logging policy chat history retention"
    },
    "search": {
      "top_k": 8,
      "filters": {"product": "llm_app", "doc_type": ["policy", "security"]},
      "latency_ms": 74
    },
    "rerank": {
      "enabled": true,
      "model": "rerank_v3",
      "top_n": 4,
      "latency_ms": 31
    },
    "results": [
      {
        "doc_id": "POL-0142",
        "chunk_id": "POL-0142#c07",
        "rank": 1,
        "score": 0.83,
        "source": "confluence",
        "last_updated": "2026-01-28",
        "token_count": 312
      }
    ],
    "recall_proxies": {
      "has_any_results": true,
      "top1_score": 0.83,
      "top4_mean_score": 0.71
    }
  },
  "context_build": {
    "strategy": "topN_chunks_then_dedupe",
    "max_context_tokens": 6000,
    "selected_chunks": ["POL-0142#c07", "POL-0199#c03", "SEC-0008#c12"],
    "dedupe": {"enabled": true, "dropped_chunks": 1},
    "final_context_tokens": 1420,
    "latency_ms": 12
  },
  "generation": {
    "provider": "openai",
    "model": "gpt-4.1-mini",
    "system_prompt_hash": "sha256:2d91...0a8c",
    "tools_available": ["policy_lookup", "ticket_create"],
    "temperature": 0.2,
    "max_output_tokens": 600,
    "usage": {
      "input_tokens": 1860,
      "output_tokens": 382,
      "cached_input_tokens": 420
    },
    "latency": {
      "ttft_ms": 410,
      "total_ms": 1280
    },
    "finish_reason": "stop"
  },
  "validation": {
    "schema": {
      "name": "answer_with_citations_v1",
      "passed": true,
      "errors": []
    },
    "policy": {
      "pii_leak_check": {"passed": true, "flags": []},
      "injection_signals": {"passed": true, "flags": ["prompt_injection_phrase_detected:false"]}
    },
    "groundedness": {
      "citation_required": true,
      "citation_coverage": 0.92,
      "citation_validity_sampled": false
    }
  },
  "tools": {
    "calls": [
      {
        "tool_name": "policy_lookup",
        "call_id": "t_001",
        "args_hash": "sha256:9a11...f02c",
        "started_at": "2026-02-15T10:32:12.910Z",
        "latency_ms": 220,
        "status": "ok",
        "http_status": 200
      }
    ],
    "loop": {
      "detected": false,
      "max_steps": 8,
      "steps_taken": 2,
      "reason": null
    }
  },
  "outcome": {
    "user_visible_status": "answered",
    "success_label": "success",
    "failure_layer": null,
    "failure_code": null,
    "support_ticket_created": false,
    "user_feedback": {
      "thumbs": null,
      "free_text": null
    }
  },
  "cost": {
    "currency": "USD",
    "estimated_total": 0.0234,
    "breakdown": {
      "retrieval": 0.0011,
      "rerank": 0.0024,
      "generation": 0.0186,
      "tool_calls": 0.0013
    },
    "cost_per_success_bucket": "0.02-0.05"
  }
}

5) How to prove "before vs after" (the only method that matters)

To avoid hand-wavy outcomes, insist on this structure:

Step 1: Define success in operational metrics

• task success rate (clear definition)
• groundedness/citation validity (sampling)
• cost per successful task
• TTFT and p95 latency
• failure rate for tool calls / loops

Step 2: Build a baseline with cohorts

Don't measure one average. Measure by: intent category, language, tenant/customer tier, document type (policy, FAQ, product spec), long-tail vs common.

Step 3: Run controlled changes

• ship one change at a time (or tag changes with versioning)
• use workload replay or a fixed eval set
• compare distributions, not just averages

Step 4: Install regression gates

• CI tests for top cohorts
• alerts for drift (quality/cost/latency)
• release checklist with stop-ship thresholds

That's how you prove impact.

Mini case studies (what "before vs after" looks like)

Note: numbers are illustrative. The structure is the point—baseline by cohort, controlled changes, and distribution-aware before/after evidence.

See our case study on low recall and inference cost reduction for real before/after examples: retrieval fixes (+18–30pt answer accuracy) and cost/latency improvements (25–60% reduction) with baseline→change→evidence.

6) Which one should you choose?

Choose a GenAI Audit if:

• you're pre-production or early pilot
• your biggest unknown is prompt structure and basic guardrails
• you don't have RAG/tools complexity yet

Choose an AI System Audit if:

• you have users in production
• quality complaints persist despite prompt tweaks
• cost/latency spiking is a problem
• you ship changes without eval gates
• you have RAG, tools, routing, or multiple models

7) Next steps