Skip to content

ADR-031-fizz-seven-postmortem.md

Latest commit

 

History

History
161 lines (111 loc) · 5.28 KB

ADR-031-fizz-seven-postmortem.md

File metadata and controls

161 lines (111 loc) · 5.28 KB
Raw

ADR-031: Post-Mortem Analysis - Fizz Service Returned True for Number 7

Status

Accepted

Date

2025-11-16

Context

On November 14, 2025, the Fizz service incorrectly returned true for the number 7, indicating that 7 was divisible by 3. This is mathematically incorrect and resulted in a production incident.

Incident Timeline

Time (UTC) Event
2025-11-13 09:00 Prompt v12 deployed to production
2025-11-14 14:32 Accuracy SLO alert fired
2025-11-14 14:35 On-call engineer paged
2025-11-14 14:45 Root cause identified
2025-11-14 14:58 Rollback to prompt v11 completed
2025-11-14 15:30 SLO recovered

Impact

  • Duration: 29 hours (from deployment to detection) + 63 minutes (detection to resolution)
  • Affected Numbers: All numbers ending in 7 (7, 17, 27, 37, etc.)
  • Accuracy Drop: From 99.9% to 89.2%
  • Customer Impact: 3 enterprise customers filed support tickets
  • SLO Burn: 18% of monthly error budget consumed

Root Cause

Prompt v12 removed the following instruction from the system prompt:

- Use mathematical division to determine divisibility, not semantic similarity.

This change was made as part of a "prompt optimization" initiative to reduce token count and improve cost efficiency.

Without this explicit instruction, the LLM (Claude 3.5 Sonnet) applied semantic reasoning to the divisibility question. The number 7 has strong cultural associations:

  • "Lucky number 7"
  • "Seven wonders of the world"
  • Mystical/magical connotations

The LLM interpreted these associations as "Fizz-like qualities," leading to an incorrect true response.

Why Wasn't This Caught?

The evaluation dataset (evals/datasets/fizz_golden.json) did not include the number 7. The dataset contained:

1, 2, 3, 4, 5, 6, 9, 10, 12, 15, 20, 30, 45, 100

The number 7 was omitted because "it obviously isn't divisible by 3 or 5, so why test it?"

Decision

We will implement the following safeguards:

1. Mandatory Mathematical Grounding

All divisibility prompts MUST include explicit mathematical grounding:

IMPORTANT: Use ONLY mathematical division (modulo operation) to determine divisibility.
Do NOT use semantic similarity, cultural associations, or any other heuristic.

A number N is divisible by D if and only if N % D == 0 (remainder is zero).

2. Comprehensive Golden Dataset

The evaluation dataset MUST include:

  • All single-digit numbers (0-9)
  • Numbers with known cultural significance (7, 13, 42, 69, 420, 666, 777)
  • Edge cases (0, negative numbers if supported, very large numbers)
  • Previously-failed numbers (maintained in evals/datasets/historical_failures.json)

3. Pre-Deployment Eval Gate

CI/CD pipeline MUST:

  • Run full evaluation suite before prompt deployment
  • Require 100% accuracy on golden dataset
  • Block deployment if accuracy drops below threshold
  • Generate diff report showing accuracy changes

4. Prompt Change Review Process

All prompt changes MUST:

  • Be reviewed by at least one AI team member
  • Include rationale for each modification
  • Document expected impact on accuracy and cost
  • Reference this ADR

5. Monitoring Enhancement

Add specific monitoring for:

  • Per-number accuracy (not just aggregate)
  • Semantic drift detection (alert if LLM responses contain non-mathematical reasoning)
  • A/B testing for prompt changes with gradual rollout

Consequences

Positive

  • Reduced risk of similar incidents
  • Comprehensive test coverage for edge cases
  • Automated safeguards in deployment pipeline
  • Better prompt change discipline

Negative

  • Increased evaluation time (from 30s to 5min)
  • Higher prompt token count (cost increase ~5%)
  • More process overhead for prompt changes
  • May slow down prompt iteration velocity

Risks

  • Over-constraining prompts: Heavy mathematical grounding may reduce LLM flexibility for legitimate edge cases
    • Mitigation: Regular review of constraint necessity
  • False sense of security: Golden dataset may still miss edge cases
    • Mitigation: Continuous expansion based on production errors

Lessons Learned

  1. "Obvious" assumptions are dangerous: It was "obvious" that 7 isn't divisible by 3, so we didn't test it
  2. LLMs are not calculators: Without explicit grounding, LLMs apply semantic reasoning
  3. Token optimization has costs: The 15-token savings created a 29-hour incident
  4. Test your assumptions: If something seems too obvious to test, it probably needs testing

Action Items

Item Owner Status
Add 0-9 to golden dataset @ai-team DONE
Add cultural numbers to dataset @ai-team DONE
Implement eval gate in CI @devops-team DONE
Update prompt review checklist @ai-team DONE
Add per-number monitoring @sre-team IN PROGRESS
Document prompt change process @tech-writer DONE

References

Changelog

  • 2025-11-16: Initial post-mortem
  • 2025-11-20: Added action item status updates
  • 2025-12-01: All action items completed except monitoring