Skip to content

fizz-service-not-fizzing.md

Latest commit

 

History

History
285 lines (218 loc) · 9.03 KB

fizz-service-not-fizzing.md

File metadata and controls

285 lines (218 loc) · 9.03 KB
Raw

Runbook: Fizz Service Not Fizzing

Last Updated: 2025-12-01 Owner: @fizz-team Severity: SEV-2 if sustained, SEV-3 if transient

Overview

This runbook covers scenarios where the Fizz service is not producing expected output. "Not Fizzing" can manifest as:

  1. Service not consuming from Kafka
  2. Service not producing to fizz-topic
  3. LLM calls failing
  4. Incorrect Fizz determinations

Prerequisites

  • Access to Grafana dashboards
  • Access to service logs (Datadog/CloudWatch)
  • kubectl access to Kubernetes cluster
  • Upstash Kafka console access
  • ANTHROPIC_API_KEY for testing

Detection

Alerts That May Fire

Alert Threshold Description
fizz_throughput_low <50/min for 5min Fizz events not being produced
fizz_consumer_lag >1000 messages for 5min Kafka consumer falling behind
fizz_error_rate >1% for 5min Errors in Fizz service
fizz_accuracy_low <99.9% for 5min Incorrect Fizz results

Quick Health Check

# Check service status
kubectl get pods -l app=fizz-service

# Check recent logs
kubectl logs -l app=fizz-service --tail=100

# Check Kafka consumer lag
curl -s "$UPSTASH_KAFKA_REST_URL/topics/number-topic/consumer-groups/fizz-service-consumer-group" \
  -H "Authorization: Basic $UPSTASH_AUTH" | jq '.lag'

# Check fizz-topic production
curl -s "$UPSTASH_KAFKA_REST_URL/topics/fizz-topic/messages?count=5" \
  -H "Authorization: Basic $UPSTASH_AUTH" | jq '.'

Diagnosis Flowchart

                    ┌─────────────────────┐
                    │ Fizz Alert Fired    │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │ Is pod running?     │
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              │ NO             │                │ YES
              ▼                │                ▼
    ┌─────────────────┐        │      ┌─────────────────┐
    │ Check pod logs  │        │      │ Is consumer     │
    │ for crash       │        │      │ lag high?       │
    └────────┬────────┘        │      └────────┬────────┘
             │                 │               │
             ▼                 │    ┌──────────┼──────────┐
    Go to: POD CRASH           │    │ YES      │          │ NO
                               │    ▼          │          ▼
                               │  ┌──────────┐ │   ┌──────────────┐
                               │  │Kafka     │ │   │ Check LLM    │
                               │  │issues    │ │   │ error rate   │
                               │  └────┬─────┘ │   └──────┬───────┘
                               │       │       │          │
                               │       ▼       │   ┌──────┼──────┐
                               │   Go to:      │   │ HIGH │      │ LOW
                               │   KAFKA LAG   │   ▼      │      ▼
                               │               │ Go to:   │   Go to:
                               │               │ LLM      │   ACCURACY
                               │               │ ERRORS   │   ISSUE
                               │               │          │
                               └───────────────┴──────────┘

Scenario: POD CRASH

Symptoms

  • Pod in CrashLoopBackOff or Error state
  • No recent fizz events

Investigation

# Get pod status
kubectl describe pod -l app=fizz-service

# Get crash logs
kubectl logs -l app=fizz-service --previous

# Common crash causes:
# - OOM: Check memory limits
# - Config error: Check environment variables
# - Dependency failure: Check Kafka/Anthropic connectivity

Resolution

Cause Resolution
OOM Killed Increase memory limit in deployment
Missing env var Check ConfigMap/Secret
Kafka connection Verify Upstash credentials
Anthropic API error Check API key validity 
# Restart pod
kubectl rollout restart deployment/fizz-service

# Scale up if needed
kubectl scale deployment/fizz-service --replicas=3

Scenario: KAFKA LAG

Symptoms

  • Consumer lag increasing
  • fizz-topic production slower than number-topic consumption
  • Pod is running but processing slowly

Investigation

# Check consumer lag trend
# In Grafana: fizzbuzz.fizz.consumer_lag

# Check processing time
# In Grafana: fizzbuzz.fizz.processing_time_p99

# Check if lag is increasing or stable
watch -n 5 'curl -s "$UPSTASH_KAFKA_REST_URL/consumer-groups/fizz-service-consumer-group" \
  -H "Authorization: Basic $UPSTASH_AUTH" | jq ".lag"'

Resolution

# Scale up consumers (if lag is increasing)
kubectl scale deployment/fizz-service --replicas=5

# If stuck on specific offset, skip messages (CAUTION)
# This will lose messages - only use if absolutely necessary
# Contact @fizz-team-lead before executing

If lag is caused by slow LLM responses, see LLM ERRORS section.

Scenario: LLM ERRORS

Symptoms

  • High error rate in logs
  • Anthropic API errors
  • Timeouts on LLM calls

Investigation

# Check Anthropic API status
curl -s https://status.anthropic.com/api/v2/status.json | jq '.status'

# Check error types in logs
kubectl logs -l app=fizz-service | grep -i error | tail -50

# Check API key validity
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model": "claude-3-5-haiku-20241022", "max_tokens": 10, "messages": [{"role": "user", "content": "Hi"}]}'

Resolution

Error Type Resolution
401 Unauthorized Rotate API key, update secret
429 Rate Limit Implement backoff, contact Anthropic for limit increase
500 Server Error Wait for Anthropic recovery, consider fallback
Timeout Increase timeout, check network 
# Update API key
kubectl create secret generic anthropic-credentials \
  --from-literal=api-key=$NEW_API_KEY \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart to pick up new key
kubectl rollout restart deployment/fizz-service

Scenario: ACCURACY ISSUE

Symptoms

  • fizz_accuracy_low alert
  • Incorrect Fizz determinations
  • Customer complaints

Investigation

# Check recent accuracy
# Query analytics database:
# SELECT
#   COUNT(*) as total,
#   SUM(CASE WHEN is_correct THEN 1 ELSE 0 END) as correct,
#   100.0 * SUM(CASE WHEN is_correct THEN 1 ELSE 0 END) / COUNT(*) as accuracy
# FROM fizzbuzz_results
# WHERE timestamp > NOW() - INTERVAL '1 hour'

# Check which numbers are failing
# Query for incorrect results

# Check current prompt version
kubectl exec -it deploy/fizz-service -- env | grep FIZZ_PROMPT_VERSION

# Compare to expected version
cat prompts/fizz-divisibility-v12.txt

Resolution

If prompt version is wrong:

# Update prompt version in ConfigMap
kubectl edit configmap fizz-service-config
# Change FIZZ_PROMPT_VERSION to correct version

# Restart to pick up change
kubectl rollout restart deployment/fizz-service

If prompt content is wrong (see ADR-031 for historical context):

# Rollback to last known good prompt
kubectl set env deployment/fizz-service FIZZ_PROMPT_VERSION=v11

# Monitor accuracy recovery
# In Grafana: fizzbuzz.fizz.accuracy

Escalation

Condition Escalate To
Unable to resolve in 30 minutes @fizz-team-lead
Customer-reported issues @fizz-team-lead + @customer-success
Anthropic API outage @ai-team-lead
Kafka infrastructure issues @infrastructure-team
Accuracy below 95% @engineering-manager (SEV-1)

Post-Incident

After resolving:

  1. Update incident channel with resolution
  2. Document timeline in incident ticket
  3. Schedule post-mortem if SEV-1 or SEV-2
  4. Update this runbook if new failure mode discovered

References

Runbook Review Schedule: Quarterly Next Review: 2026-03-01 Feedback: #fizz-team or update this document directly