[SL_CORE]
< CD ..||6 min read

Testing LLM Applications: From Evals to Prod Reliability

# LLM apps can't be unit tested like regular code — outputs are probabilistic. Learn the eval-driven testing approach: golden datasets, LLM-as-judge, regression suites, and CI integration that actually catches failures.

[testing][llm][reliability][production]

You can't write assertEquals("Paris", llm.answer("Capital of France?")) — that's too brittle. LLM testing requires a different approach: eval-driven development, where you measure quality across a representative dataset and catch regressions before they reach users. Here's how to build this properly.

The Eval-Driven Approach

Traditional unit tests check exact outputs. LLM evals check output quality across many examples:

Unit test: input X → exactly output Y
LLM eval: 100 inputs → average quality ≥ threshold

The three components:

  1. Golden dataset — representative inputs with known-good outputs
  2. Scoring function — measures quality of new outputs vs. expected
  3. Threshold — minimum acceptable score before deployment

Building a Golden Dataset

A golden dataset is your most valuable testing asset. Build it deliberately:

interface EvalExample {
  id: string;
  input: string | Record<string, unknown>;
  expectedOutput?: string;           // for exact-match tasks
  expectedCriteria?: string[];       // for open-ended tasks
  category: string;                  // for grouping results
  difficulty: "easy" | "medium" | "hard";
  notes?: string;                    // why this case matters
}

const CODE_REVIEW_EVAL_SET: EvalExample[] = [
  {
    id: "basic-sql-injection",
    input: { code: "const query = `SELECT * FROM users WHERE id = ${userId}`", language: "typescript" },
    expectedCriteria: [
      "Identifies SQL injection vulnerability",
      "Suggests parameterized queries",
      "Rates severity as critical or high",
    ],
    category: "security",
    difficulty: "easy",
  },
  {
    id: "subtle-race-condition",
    input: { code: `// check-then-act race condition in distributed system...`, language: "go" },
    expectedCriteria: [
      "Identifies race condition",
      "Mentions distributed systems context",
      "Suggests atomic operations or locks",
    ],
    category: "correctness",
    difficulty: "hard",
  },
  // ... 50–200 more examples
];

Aim for 50–200 examples per task type. Include:

  • Typical cases (50%)
  • Edge cases (30%)
  • Adversarial/tricky cases (20%)

Scoring Functions

LLM-as-Judge (most flexible)

async function llmJudge(
  input: string,
  output: string,
  criteria: string[]
): Promise<{ score: number; passed: string[]; failed: string[]; reasoning: string }> {
  const criteriaList = criteria.map((c, i) => `${i + 1}. ${c}`).join("\n");

  const response = await anthropic.messages.create({
    model: "claude-opus-4-8", // use strongest model as judge
    max_tokens: 500,
    system: "You are an objective evaluator. Score outputs strictly against criteria.",
    messages: [{
      role: "user",
      content: `Input: ${input}
Output: ${output}

Does the output satisfy each criterion?
${criteriaList}

Return JSON: {
  "results": [{"criterion": "...", "passed": true/false, "reason": "brief"}],
  "overallScore": 0.0-1.0,
  "reasoning": "summary"
}`,
    }],
  });

  const { results, overallScore, reasoning } = JSON.parse(response.content[0].text);
  return {
    score: overallScore,
    passed: results.filter((r: any) => r.passed).map((r: any) => r.criterion),
    failed: results.filter((r: any) => !r.passed).map((r: any) => r.criterion),
    reasoning,
  };
}

Deterministic Scoring (for structured outputs)

For tasks with structured outputs, check programmatically:

function scoreSQLOutput(output: string, example: EvalExample): number {
  let score = 0;
  const checks = [
    { test: () => /SELECT|INSERT|UPDATE|DELETE/i.test(output), weight: 0.3 },
    { test: () => !output.includes("DROP TABLE"), weight: 0.2 },
    { test: () => /WHERE\s+\w+\s*=\s*\?/i.test(output), weight: 0.3 }, // parameterized
    { test: () => output.trim().endsWith(";"), weight: 0.2 },
  ];

  for (const { test, weight } of checks) {
    if (test()) score += weight;
  }
  return score;
}

Mix both: use deterministic scoring for measurable properties and LLM-as-judge for semantic quality.

Running Evals

interface EvalResult {
  exampleId: string;
  category: string;
  score: number;
  passed: boolean;
  output: string;
  latencyMs: number;
  costUSD: number;
}

async function runEvalSuite(
  examples: EvalExample[],
  modelUnderTest: string,
  taskFunction: (input: unknown) => Promise<string>,
  passThreshold = 0.7
): Promise<EvalReport> {
  const results: EvalResult[] = [];

  // Run in parallel with concurrency limit
  const sem = new Semaphore(10);
  await Promise.all(
    examples.map(async (example) => {
      await sem.acquire();
      const start = Date.now();

      try {
        const output = await taskFunction(example.input);
        const { score } = await llmJudge(
          JSON.stringify(example.input),
          output,
          example.expectedCriteria ?? []
        );

        results.push({
          exampleId: example.id,
          category: example.category,
          score,
          passed: score >= passThreshold,
          output,
          latencyMs: Date.now() - start,
          costUSD: 0, // tracked separately via tracing
        });
      } catch (err) {
        results.push({
          exampleId: example.id,
          category: example.category,
          score: 0,
          passed: false,
          output: `ERROR: ${err}`,
          latencyMs: Date.now() - start,
          costUSD: 0,
        });
      } finally {
        sem.release();
      }
    })
  );

  return buildReport(results, modelUnderTest, passThreshold);
}

CI Integration

Add evals to your CI pipeline so regressions block deployment:

// eval-runner.ts — runs in CI
async function ciEvalCheck(): Promise<void> {
  const PASS_THRESHOLD = 0.85;  // 85% of examples must pass
  const CATEGORY_THRESHOLDS: Record<string, number> = {
    security: 0.95, // security checks must be near-perfect
    correctness: 0.90,
    performance: 0.80,
  };

  const report = await runEvalSuite(EVAL_EXAMPLES, MODEL, myTask, PASS_THRESHOLD);

  console.log(`Overall pass rate: ${(report.passRate * 100).toFixed(1)}%`);
  console.log(`By category:`, report.byCategory);

  // Check category thresholds
  const failures: string[] = [];
  for (const [category, threshold] of Object.entries(CATEGORY_THRESHOLDS)) {
    const categoryRate = report.byCategory[category]?.passRate ?? 0;
    if (categoryRate < threshold) {
      failures.push(`${category}: ${(categoryRate * 100).toFixed(1)}% < ${(threshold * 100).toFixed(1)}% required`);
    }
  }

  if (failures.length > 0 || report.passRate < PASS_THRESHOLD) {
    console.error("EVAL FAILED:\n" + failures.join("\n"));
    process.exit(1); // fail CI
  }

  console.log("All evals passed ✓");
}

Add to your GitHub Actions:

# .github/workflows/eval.yml
- name: Run LLM Evals
  run: npx ts-node eval-runner.ts
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Shadow Testing in Production

Test new model versions against prod traffic without user impact:

async function shadowTest(
  request: UserRequest,
  currentModel: string,
  candidateModel: string
): Promise<string> {
  // Run current model — user gets this response
  const [current, candidate] = await Promise.all([
    runModel(request, currentModel),
    runModel(request, candidateModel), // shadow — user never sees this
  ]);

  // Log comparison for analysis
  await logComparison({
    requestId: request.id,
    currentModel,
    candidateModel,
    currentOutput: current,
    candidateOutput: candidate,
    // Async LLM judge evaluates both outputs later
  });

  return current; // always return current to user
}

Run shadow tests for 1 week, collect 10K comparisons, then use the LLM judge to score both. If candidate wins by >5%, promote to production.

FAQ

How many eval examples do I need? Minimum viable: 20 (catches obvious regressions). Good coverage: 100. Comprehensive: 500+. Quality matters more than quantity — 50 well-crafted examples beat 500 random ones.

How do I prevent eval set contamination? Don't include eval examples in your training data or system prompt. Keep the eval set in a separate repository, access-controlled, and never shown to the model being evaluated.

What's an acceptable pass rate? Depends on the task. For safety-critical checks: 95%+. For creative tasks: 70%+ is fine. For customer-facing responses: aim for 85%+. Set thresholds based on the cost of failures in your domain.

How often should I run evals? On every prompt change (mandatory), on every model upgrade, and weekly for production health checks. Evals in CI are blocking; weekly evals are advisory.

What do I do when evals fail? Examine failing examples, identify the pattern, fix the prompt or add examples to training. Never just raise the threshold to make CI pass — that defeats the purpose.