Audit Readiness

Minimum Logging and Tracing Before You Pay for an Audit

A production AI audit fails for one reason more than any other: you don't have enough observable evidence to diagnose what's actually breaking. Without the right logs and traces, every audit becomes a guessing game.

To understand what you keep after the audit, read AI System Audit Deliverables. For an Audit Readiness Check, see our AI System Audit service.

What it means

Audit readiness is not "we have logs"

It means you can reliably answer, for a representative set of production requests:

Question 1

What happened? (inputs → retrieval → tools → output)

Question 2

Where did it fail? (retrieval vs generation vs tools vs policy)

Question 3

Why did it fail? (root causes you can see in evidence)

Question 4

What did it cost and how long did it take? (per stage)

Question 5

What changed? (prompt/model/routing/version)

If you can't answer those, a vendor can still help—but the engagement will shift from "audit" to "instrumentation project."

The audit packet

Minimum logging/tracing schema (5 layers)

Each request produces an audit packet. You don't need to log everything—just the right things consistently.

Layer 1(required)

Request metadata

Compare versions, enable cohort-based analysis

request_idtimestampenvironmentapp_versionprompt_versionmodel_iduser_cohort_labelsinput_length

Layer 2(if RAG)

Retrieval evidence

Identify recall vs ranking vs context construction failure

candidates (doc_id, chunk_id, score, rank)rerank scores (if used)selected chunks + token count

Layer 3(required)

Generation evidence

Cost decomposition, quality signals, regression detection

model_idinput/output tokensstop_reasonvalidation (schema/citation/policy)

Layer 4(if tools)

Tool / agent evidence

Separate LLM slowness from tool timeouts; detect loop storms

tool_calls_countloop_countper-call: name, latency, status, retries

Layer 5(required)

Timing + cost breakdown

Find hidden spenders, explain P95 spikes

stage latencies (retrieval, rerank, gen, tools)ttft_mstotal_usdretry/timeout counts

Privacy & compliance

"Safe logging" modes

Audit readiness must not create a compliance nightmare. Choose one:

Mode A

Full-fidelity

Raw prompts + text. Internal only, strict access + retention.

Mode BRecommended

Token-safe + reference

Prompt hash, chunk IDs + scores, redacted response. 80% diagnostic power, far less risk.

Mode C

Proof-only

IDs + scores + validation + metrics. No raw text. Slower debugging.

Starting from zero

The minimum 10 things

If you only implement 10 things, implement these:

request_id + timestamps
prompt/model/routing versions (hashes)
cohort labels (intent/lang/tenant)
retrieval candidate IDs + scores
final selected chunk IDs
tool call logs (name, latency, status, retries)
stage latency breakdown
token counts (in/out)
validation signals (schema/citation/policy)
retry/timeout counts

With these, an audit can reliably produce: baseline scorecard, failure taxonomy, cost + latency teardown, root-cause attribution and roadmap.

Avoid these

Common mistakes

Logging only the final answer—can't diagnose retrieval vs generation vs tools
No versioning for prompts and routing—can't attribute regressions
No cohort labels—everything averages out, real failures hide
Logging too much then turning it off—trends disappear
No retention/access policy—auditors blocked or compliance blocks everything

Implementation

Quick and practical (7 days)

A practical week-long implementation guide.

Day 1–2

Instrumentation

request_id propagation
stage timers
token counts + versions

Day 3–4

Retrieval + tools

candidate chunk IDs + scores
selected chunks
tool call details

Day 5

Storage + privacy

Choose mode (A/B/C)
redaction + retention
RBAC

Day 6–7

First baseline

Sample 200–500 requests
compute baseline
tag top 10 failures

FAQ

Common questions

What if we don't have any logging yet?

Start with the minimum 10. Day 1–2: request_id, stage timers, token counts. Day 3–4: retrieval + tool logs. Day 5: storage + privacy. Day 6–7: first baseline sample.

Which safe logging mode should we use?

Mode B (token-safe + reference) gives 80% diagnostic power with far less risk. Many teams run Mode B in production and temporarily enable full-fidelity during incident windows.

Can we do an audit without this schema?

Yes, but the engagement shifts from 'audit' to 'instrumentation project.' You'll spend time building observability before you can produce a baseline and failure taxonomy.

What's an Audit Readiness Check?

We review your anonymized traces and pipeline diagram, then tell you what schema fields you're missing, what to instrument first for biggest ROI, and how to keep it privacy-safe.

Share a few anonymized traces and your pipeline diagram. We'll tell you what schema fields you're missing. See our AI System Audit service to request an Audit Readiness Check.