Skip to content

ADR-047-llm-evaluation-strategy.md

Latest commit

 

History

History
237 lines (166 loc) · 7.8 KB

ADR-047-llm-evaluation-strategy.md

File metadata and controls

237 lines (166 loc) · 7.8 KB
Raw

ADR-047: LLM Evaluation Strategy

Status

Accepted

Date

2026-01-30

Context

FizzBuzz Enterprise Edition relies on large language models for divisibility determination. Following INC-2025-1114-001, where the Fizz service incorrectly classified 7 as divisible by 3, the FizzBuzz Reliability Council mandated continuous evaluation of LLM accuracy.

The Accuracy Gap

The incident revealed that we had no systematic way to validate LLM behavior before deployment. Our only validation was manual testing by developers, which failed to catch the regression introduced in prompt version 12.

Post-incident analysis identified the following gaps:

  • EVAL-AUDIT-2025-001: No automated accuracy validation for LLM outputs
  • EVAL-AUDIT-2025-002: Golden datasets existed but were not integrated into any pipeline
  • EVAL-AUDIT-2025-003: No regression detection mechanism for prompt changes
  • EVAL-AUDIT-2025-004: Model comparison data collected manually and inconsistently

Root Cause Analysis

INC-2025-1114-001 occurred because prompt v12 included the phrase "consider the mathematical and cultural significance" which caused the model to associate 7 with "lucky number" semantics rather than performing strict modulo arithmetic. This semantic confusion was not caught because:

  1. Manual testing used "obvious" numbers (3, 6, 9, 15) that would pass regardless
  2. No regression suite existed for historically problematic inputs
  3. The prompt change was reviewed for style but not evaluated for accuracy

Business Impact

The incident resulted in:

  • 847 incorrect FizzBuzz determinations over 23 minutes
  • 3 enterprise customer escalations
  • $12,400 in SLA credits
  • Mandatory incident review with the VP of Engineering

Decision

We will implement a comprehensive LLM evaluation framework using promptfoo, an open-source evaluation toolkit.

Why promptfoo?

We evaluated four LLM evaluation frameworks:

Option 1: Anthropic Evals (anthropics/evals)

Anthropic's internal evaluation framework, open-sourced for the community.

Pros:

  • Native integration with Claude models
  • Used by Anthropic for their own model development
  • Strong support for constitutional AI evaluations

Cons:

  • Python-only, requires separate toolchain from our TypeScript services
  • Focused on model development, not prompt engineering
  • Limited assertion types for structured output validation
  • No built-in model comparison features

Option 2: OpenAI Evals (openai/evals)

OpenAI's evaluation framework for GPT models.

Pros:

  • Large community and example library
  • Good documentation

Cons:

  • OpenAI-centric, Anthropic support is community-maintained
  • Python-only
  • Heavyweight for our use case (designed for model training feedback)

Option 3: promptfoo

Open-source prompt engineering and evaluation toolkit.

Pros:

  • TypeScript/JavaScript native (matches our stack)
  • Provider-agnostic (Anthropic, OpenAI, local models)
  • Built-in model comparison and regression detection
  • YAML configuration (easy to review in PRs)
  • Local caching reduces API costs during development
  • Active development (47% growth in GitHub stars in 2025)

Cons:

  • Smaller community than OpenAI Evals
  • Some advanced features require paid cloud tier

Option 4: Custom Evaluation Framework

Build our own evaluation system.

Pros:

  • Complete control over implementation
  • No external dependencies

Cons:

  • Significant engineering investment
  • Maintenance burden
  • Would duplicate existing open-source solutions

Decision Matrix

Criteria Weight Anthropic Evals OpenAI Evals promptfoo Custom
Language match (TypeScript) 25% 1 1 5 5
Provider agnostic 20% 2 2 5 5
Ease of integration 20% 3 3 5 2
Model comparison 15% 3 4 5 3
Community/Support 10% 4 5 4 1
Maintenance burden 10% 4 4 4 1
Weighted Score 2.35 2.50 4.85 3.15

promptfoo was selected based on language alignment, provider flexibility, and low integration overhead.

Implementation

Package Structure

Evaluations are implemented as a workspace package (@fizzbuzz-enterprise/evals) to maintain separation of concerns and enable independent versioning.

packages/evals/
├── datasets/
│   ├── fizz_golden.json     # Version-controlled expected outputs
│   └── buzz_golden.json
├── results/
│   └── model_comparison_*.json  # Historical baselines
├── src/
│   ├── compare.js           # Regression detection
│   └── report.js            # Multi-format reporting
├── promptfooconfig.yaml     # Combined evaluation
├── promptfooconfig.fizz.yaml
└── promptfooconfig.buzz.yaml

Test Categories

Category Purpose Example Numbers
single_digit Foundational accuracy 0-9
basic Common cases 12, 15, 18, 21
cultural Semantic confusion prevention 42, 69, 420, 666
regression Historical incident cases 7, 77, 777
large Mathematical reasoning at scale 123456, 999999

Accuracy Thresholds

Metric Threshold Rationale
Minimum accuracy 95% Below this, prompt requires revision
Production accuracy 100% Required for deployment approval
Regression tolerance 2% Triggers mandatory review

CI Integration

Evaluation configuration is validated in CI without making LLM API calls:

  • promptfoo validate runs on every PR touching packages/evals/ or prompts/
  • Ensures YAML syntax is correct and test cases are well-formed
  • No API costs incurred during CI

Full evaluation runs are triggered manually or on a weekly schedule via separate workflow.

Consequences

Positive

  • Automated Regression Detection: Prompt changes are validated against 85+ test cases
  • Model Comparison: Can evaluate cost/accuracy tradeoffs when considering model changes
  • Audit Trail: All evaluation results are stored for compliance review
  • Developer Confidence: Clear pass/fail criteria for prompt engineering
  • Cost Visibility: Per-evaluation cost tracking enables budget planning

Negative

  • API Costs: Full evaluation suite costs ~$0.02 per run
  • Additional Dependency: promptfoo added to workspace dependencies
  • Learning Curve: Team must learn promptfoo configuration syntax

Risks

Risk Likelihood Impact Mitigation
promptfoo project abandoned Low Medium Configs are portable YAML, can migrate
Test suite becomes stale Medium High Quarterly review of test coverage
False confidence from passing tests Medium Medium Continuously add cases from production issues

Operational Considerations

Running Evaluations

# Full evaluation (requires ANTHROPIC_API_KEY)
pnpm eval

# Service-specific
pnpm eval:fizz
pnpm eval:buzz

# View results
pnpm eval:view

Adding Test Cases

When a production issue is identified:

  1. Add the failing case to the appropriate golden dataset
  2. Add to promptfoo config with category: regression
  3. Document the incident reference in the test description
  4. Verify the fix passes the new test case

Cost Management

  • Local development uses cached responses by default
  • CI runs validation only (no API calls)
  • Weekly scheduled runs use --no-cache for fresh data
  • Cost alerts configured for >$1/day evaluation spend

References

Changelog

  • 2026-01-30: Initial ADR