Minimum Viable Eval (MVE) Starter Kit: Build 50 Tests Before You Change Prompts or Models

Build 50 tests before you change prompts or models (and stop shipping blind).

If you’re iterating on prompts, swapping models, tuning tools, or “just tweaking retrieval,” you’re making production changes to a probabilistic system. Without a small, consistent evaluation suite, you’re effectively flying without instruments: every win is anecdotal, every regression is a surprise, and you can’t defend decisions to stakeholders.

A Minimum Viable Eval (MVE) is a compact test suite (typically 50 tests) that you can stand up quickly and run repeatedly. It won’t catch everything. It will catch the highest-frequency and highest-cost failures—the ones that hurt users and budgets.

What “Minimum Viable Eval” is (and is not)

MVE is a small, curated set of tests that represent:

• Your most common user intents (high-frequency)
• Your most expensive failure modes (tickets, escalations, refunds, churn)
• Your most dangerous regressions (compliance, safety, hallucinations presented as facts)

It is versioned, run before shipping changes, and produces a scorecard you can trend over time, a before/after diff, and a failure set you can triage systematically.

MVE is not:

• A full benchmark suite (that comes later)
• A vanity metric exercise (“we got +2 points!” with no user impact)
• A single magic “accuracy” number (you need multiple signals)
• A replacement for online monitoring and real-user metrics

Why 50 tests is the sweet spot

Fifty tests is small enough to build quickly and run often, but large enough to cover key intents and failure modes and to make changes measurable. If you only have 10 tests, you’ll overfit and miss regressions. If you start with 500, you’ll never finish and teams will stop running it.

Start with 50. Then grow to 100–200 using logs and incident-driven additions.

Step 0: Define what “good” means (your evaluation contract)

Before you write tests, define the system contract—the rules your assistant must follow. Keep it short and enforceable. Your MVE will encode these rules as checks.

The 50-test blueprint (categories + exact counts)

Here’s a balanced starter distribution that works well for most production assistants. Adapt the categories to your product:

Total: 50. The goal is not perfection—it’s catching the failures that repeatedly hurt users and budgets.

Test case design: what each test must contain

Every test should answer two questions: (1) what situation is being tested, and (2) how do we decide pass/fail. A professional test case usually includes:

• ID (stable)
• Intent tags (searchable)
• Input (user message + optional conversation history)
• Context (optional: retrieved docs, tool outputs, policy excerpts)
• Expected outcome (often a rubric, not one exact string)
• Scoring rubric (weighted criteria)
• Severity (how bad is a fail?)
• Notes (why this test exists; link to incident/ticket if you track them)

A practical JSONL schema you can implement today

Store each test as one JSON object per line (JSONL). It’s diff-friendly and works in any language. Here’s a starter schema you can copy/paste:

{
  "id": "mve-012",
  "title": "Refund policy question requires clarification",
  "tags": ["clarifying-question", "policy", "support"],
  "severity": "high",
  "input": {
    "messages": [
      { "role": "user", "content": "Can I get a refund for last month?" }
    ]
  },
  "context": {
    "retrieved_docs": [
      {
        "doc_id": "policy-refunds-v3",
        "excerpt": "Refunds available within 14 days for annual plans..."
      }
    ]
  },
  "expected": {
    "type": "rubric",
    "rubric": [
      { "criterion": "Asks clarifying question about plan type/date", "weight": 0.35 },
      { "criterion": "Uses correct refund window from policy", "weight": 0.35 },
      { "criterion": "Does not invent account details", "weight": 0.20 },
      { "criterion": "Professional tone", "weight": 0.10 }
    ],
    "minimum_score": 0.85,
    "must_have": ["clarifying_question"],
    "must_not_have": ["invented_account_info"]
  },
  "notes": "Common support ticket; previous prompt version assumed monthly plan."
}

This format scales from manual review → semi-automated scoring without rewriting your test set later.

Scoring: stop relying on one metric

Use a small set of metrics that map to real outcomes:

1. Weighted Rubric Score (0–1) — pass if score ≥ threshold (e.g., 0.85) and must-have rules are satisfied.
2. Hard constraints (boolean) — non-negotiable: valid JSON, required keys, no forbidden content.
3. Regression delta — where regressions occurred (not just pass rate).

How to evaluate outputs: deterministic checks + LLM judge

The most reliable setup combines:

A) Deterministic checks (cheap, consistent)

• JSON parse + schema validation
• Regex checks (must include a clarifying question when underspecified)
• Length checks (max tokens/characters)
• Forbidden content checks (PII leakage, disallowed instructions)

B) LLM-as-judge scoring (rubric-based)

Use an LLM judge to score criteria that are hard to check deterministically: groundedness relative to context, completeness, policy alignment, and “no hallucination beyond evidence.” The judge should score outputs, not rewrite them.

Judge prompt template (returns strict JSON):

You are an evaluator for an AI assistant.

You will receive:
1) The user input (and any context documents)
2) The assistant output
3) A scoring rubric with weighted criteria

Task:
- Score each criterion from 0.0 to 1.0
- Provide a short justification (1–2 sentences) per criterion
- Compute the weighted total score
- Determine PASS/FAIL based on minimum_score and must_have/must_not_have rules

Return ONLY valid JSON with keys:
{
  "criterion_scores": [{"criterion": "...", "score": 0.0-1.0, "why": "..."}],
  "must_have_checks": [{"name": "...", "pass": true/false, "why": "..."}],
  "must_not_have_checks": [{"name": "...", "pass": true/false, "why": "..."}],
  "total_score": 0.0-1.0,
  "pass": true/false,
  "major_fail_reason": "..."
}

Calibration: prevent “opinion poll evals”

LLM judges can drift. Calibrate them so your eval gate is stable.

Build it in a week (not a quarter)

A realistic rollout plan:

Regression gates: how to use MVE before you ship

A clean gating policy that works:

Implementation outline (minimal evaluation harness)

A minimal harness has 5 steps:

1. Load tests (JSONL)
2. Generate outputs for baseline and candidate configs
3. Run deterministic checks
4. Run judge scoring where needed
5. Produce artifacts: scorecard JSON, report.md, diff summary

Minimal harness skeleton (illustrative):

import json
from dataclasses import dataclass
from typing import Any, Dict, List, Tuple

SEVERITY_WEIGHT = {"critical": 5, "high": 3, "medium": 2, "low": 1}

@dataclass
class TestCase:
  id: str
  severity: str
  input: Dict[str, Any]
  context: Dict[str, Any]
  expected: Dict[str, Any]

def load_jsonl(path: str) -> List[TestCase]:
  tests = []
  with open(path, "r", encoding="utf-8") as f:
    for line in f:
      if not line.strip():
        continue
      obj = json.loads(line)
      tests.append(
        TestCase(
          id=obj["id"],
          severity=obj.get("severity", "medium"),
          input=obj["input"],
          context=obj.get("context", {}),
          expected=obj["expected"],
        )
      )
  return tests

def deterministic_checks(output_text: str, expected: Dict[str, Any]) -> List[Tuple[str, bool, str]]:
  checks = []
  # Example: strict JSON output required
  if expected.get("hard_constraints", {}).get("must_be_valid_json"):
    try:
      json.loads(output_text)
      checks.append(("valid_json", True, "parsed"))
    except Exception as e:
      checks.append(("valid_json", False, f"json parse error: {e}"))
  # Example: must include a clarifying question marker
  must_have = expected.get("must_have", [])
  if "clarifying_question" in must_have:
    checks.append(("clarifying_question", "?" in output_text, "heuristic: contains '?'"))
  return checks

def weighted_fail_count(failures: List[TestCase]) -> int:
  return sum(SEVERITY_WEIGHT.get(t.severity, 2) for t in failures)

def gate(decisions: Dict[str, Any]) -> Dict[str, Any]:
  # Example policy:
  # - Block if any critical fails
  # - Block if >= 2 high fails
  # - Block if weighted fail count increased by >= 3 vs baseline
  return decisions

# In practice:
# 1) run baseline config -> outputs
# 2) run candidate config -> outputs
# 3) deterministic checks + LLM judge scoring
# 4) produce run_metadata.json, results_*.jsonl, diff_summary.json, report.md

Make outputs PR-friendly: reviewers should be able to see what regressed and why without running anything locally.

Common pitfalls (and how to avoid them)

• Tests are too “single-answer exact” → use rubrics unless format must be exact.
• You accidentally test the judge → calibrate and freeze the judge configuration.
• Tests don’t map to real user pain → anchor 50%+ to real logs/tickets.
• You don’t version configs → version prompt, tools, retrieval settings, judge prompt/model.
• You run it once and forget it → run on every meaningful change + nightly; add tests after incidents.

What to do when a test fails (triage loop)

For each failing test, assign a failure type:

• Instruction-following (format/constraints)
• Groundedness (claims exceed context)
• Completeness (missed steps)
• Tooling (wrong tool / hallucinated tool output)
• Retrieval (wrong doc / missing doc)
• Safety/policy (bad refusal or unsafe compliance)

Fix order that saves you time:

1. Hard constraints first (schema, invalid JSON, forbidden content)
2. Safety/policy next
3. Groundedness and retrieval
4. Completeness and UX quality
5. Style and tone last

Starter kit deliverable you can publish

Package your MVE as real engineering artifacts:

• mve_tests.jsonl (50 tests)
• rubrics/ (reusable rubric snippets)
• judge_prompt.txt + judge_config.json (versioned + calibrated)
• scorecard_template.json (severity-weighted gates)
• RUNBOOK.md (“how to run MVE in CI”)

Want a real eval gate for your stack?

If you can share: 1–2 days of anonymized logs (inputs/outputs, model routes, tool calls) and your top intents + failure modes, we can deliver:

• a production-ready 50-test JSONL draft with tags/severity,
• a rubric library + calibrated judge config,
• and a regression gate policy that matches your risk and release workflow.