Skip to content

buzz-latency-exceeded.md

Latest commit

 

History

History
229 lines (169 loc) · 5.94 KB

buzz-latency-exceeded.md

File metadata and controls

229 lines (169 loc) · 5.94 KB
Raw

Runbook: Buzz Latency Exceeded

Last Updated: 2025-12-01 Owner: @buzz-team Severity: SEV-3 (SEV-2 if SLO breached)

Overview

This runbook covers scenarios where the Buzz service latency exceeds acceptable thresholds. High latency affects end-to-end FizzBuzz processing time and can breach SLOs.

SLO Targets

Percentile Target Alert Threshold
p50 <200ms >300ms
p95 <500ms >750ms
p99 <1000ms >1500ms

Prerequisites

  • Access to Grafana dashboards
  • Access to service logs
  • kubectl access
  • Understanding of Anthropic API latency characteristics

Detection

Alerts

Alert Threshold Severity
buzz_latency_p50_high >300ms for 5min Warning
buzz_latency_p99_high >1500ms for 5min Critical
buzz_latency_slo_breach p99 >1000ms for 15min SEV-2

Quick Check

# Check current latency percentiles (from Prometheus)
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(buzz_processing_duration_bucket[5m]))"

# Check Anthropic API latency specifically
kubectl logs -l app=buzz-service | grep "llm_latency" | tail -20 | jq '.latency_ms'

Diagnosis

Latency Breakdown

Buzz service latency consists of:

Total Latency = Kafka Read + Validation + LLM Call + Kafka Write
                  ~10ms      ~1ms        ~150ms      ~20ms
                                           ↑
                                     Main variable

The LLM call dominates latency. High latency is usually caused by:

  1. Anthropic API slowdown: Their infrastructure issues
  2. Request queuing: Too many concurrent requests
  3. Network issues: Connectivity to Anthropic
  4. Prompt size: Larger prompts = higher latency
  5. Resource constraints: CPU/memory pressure on pods

Investigation Steps

1. Check Anthropic API Latency

# Test direct Anthropic latency
time curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-3-5-haiku-20241022",
    "max_tokens": 50,
    "messages": [{"role": "user", "content": "Is 10 divisible by 5? Reply only with JSON: {\"isDivisible\": true/false}"}]
  }'

# Expected: <500ms
# If >1000ms: Anthropic issue

2. Check Service Metrics

# In Grafana, check these panels:
# - buzz_llm_latency_p99
# - buzz_processing_time_breakdown
# - buzz_concurrent_requests

# Or query Prometheus directly:
curl "http://prometheus:9090/api/v1/query?query=buzz_concurrent_llm_requests"

3. Check Resource Utilization

# Check pod resources
kubectl top pods -l app=buzz-service

# Check for throttling
kubectl describe pod -l app=buzz-service | grep -A5 "Limits:"

4. Check Prompt Cache Hit Rate

# Low cache hit rate = higher latency
kubectl logs -l app=buzz-service | grep "cache_hit" | tail -100 | \
  jq -s 'group_by(.cache_hit) | map({hit: .[0].cache_hit, count: length})'

# Expected hit rate: >80%

Resolution

Scenario: Anthropic API Slow

Symptoms: Direct API test shows >500ms latency

Resolution:

  1. Check Anthropic Status Page
  2. If degradation reported, wait for recovery
  3. If no degradation reported, contact Anthropic support

Mitigation while waiting:

# Increase timeout to avoid errors
kubectl set env deployment/buzz-service LLM_TIMEOUT_MS=5000

# Consider switching to faster model (if acceptable accuracy trade-off)
# NOTE: Requires approval from @ai-team-lead
kubectl set env deployment/buzz-service ANTHROPIC_MODEL=claude-3-5-haiku-20241022

Scenario: Request Queuing

Symptoms: High concurrent request count, latency increases with load

Resolution:

# Scale up to handle load
kubectl scale deployment/buzz-service --replicas=5

# Check if autoscaler is working
kubectl get hpa buzz-service-hpa

# Manually adjust if needed
kubectl patch hpa buzz-service-hpa -p '{"spec":{"maxReplicas":10}}'

Scenario: Low Cache Hit Rate

Symptoms: Cache hit rate <70%, latency higher than baseline

Investigation:

# Check if prompt changed recently
kubectl exec -it deploy/buzz-service -- env | grep BUZZ_PROMPT_VERSION

# Check cache TTL alignment with traffic
# If traffic is bursty, cache may expire between bursts

Resolution:

  • Ensure steady request flow to maintain cache
  • Consider implementing local request coalescing
  • Review prompt for cache-unfriendly changes

Scenario: Resource Constraints

Symptoms: High CPU utilization, possible throttling

Resolution:

# Increase resource limits
kubectl patch deployment buzz-service -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "buzz-service",
          "resources": {
            "limits": {"cpu": "1000m", "memory": "512Mi"},
            "requests": {"cpu": "500m", "memory": "256Mi"}
          }
        }]
      }
    }
  }
}'

Latency Optimization Checklist

If latency is chronically high, review:

  • Prompt size (smaller = faster)
  • Model selection (Haiku faster than Sonnet)
  • Prompt caching enabled and working
  • Connection pooling configured
  • Appropriate concurrency limits
  • Pod resource allocation
  • Network path to Anthropic (region proximity)

Escalation

Condition Escalate To
Unable to resolve in 30 minutes @buzz-team-lead
SLO breach sustained >1 hour @engineering-manager
Anthropic outage suspected @ai-team-lead
Requires model/prompt change @ai-team-lead

References

Runbook Review Schedule: Quarterly Next Review: 2026-03-01