Skip to content

SLO.md

Latest commit

 

History

History
208 lines (148 loc) · 5.12 KB

SLO.md

File metadata and controls

208 lines (148 loc) · 5.12 KB
Raw

Service Level Objectives (SLOs)

Overview

This document defines the Service Level Objectives for FizzBuzz Enterprise Edition. These SLOs represent our commitment to reliability and accuracy.

SLO Definitions

SLO-1: FizzBuzz Accuracy

Target: 99.9%

FizzBuzz determinations must be mathematically correct.

Metric Target Measurement Window
Accuracy 99.9% 30-day rolling

Calculation:

accuracy = (correct_fizzbuzz_results / total_fizzbuzz_results) * 100

What counts as correct:

  • Number divisible by 3 → "Fizz"
  • Number divisible by 5 → "Buzz"
  • Number divisible by 15 → "FizzBuzz"
  • Number not divisible by 3 or 5 → The number itself

Historical Performance:

Period Accuracy Notes
2025-Q4 99.94%
2025-Q3 99.91%
2025-Q2 99.87% Below target due to incident 2025-11-14
2025-Q1 99.93%

SLO-2: End-to-End Latency

Target: p99 < 2000ms

The time from number submission to FizzBuzz result availability.

Percentile Target Measurement Window
p50 < 500ms 7-day rolling
p95 < 1000ms 7-day rolling
p99 < 2000ms 7-day rolling

Breakdown by component (p50):

  • Number ingestion: ~10ms
  • Kafka publish: ~20ms
  • Fizz service (including LLM): ~200ms
  • Buzz service (including LLM): ~200ms
  • Aggregator join: ~50ms
  • Total: ~480ms

SLO-3: Availability

Target: 99.9% (three nines)

The API service must be available to serve requests.

Metric Target Measurement Window
Uptime 99.9% 30-day rolling

Error Budget:

  • 30 days × 24 hours × 60 minutes = 43,200 minutes
  • 0.1% error budget = 43.2 minutes of allowed downtime per month

Historical Performance:

Period Uptime Downtime
2025-12 99.97% 12 min
2025-11 99.89% 47 min (incident 2025-11-14)
2025-10 99.98% 8 min

SLO-4: Throughput

Target: 1000 numbers/second sustained

The system must handle the specified throughput without degradation.

Metric Target Measurement Window
Sustained throughput 1000/s 1 hour
Peak throughput 5000/s 1 minute

SLO-5: Data Freshness

Target: Analytics data < 15 minutes stale

FizzBuzz events should be queryable in the analytics layer within 15 minutes.

Metric Target Measurement Window
Event-to-query latency < 15 min 24 hours

Error Budget Policy

Budget Consumption Thresholds

Budget Remaining Action
> 50% Normal operations
25-50% Increased monitoring, defer risky deployments
10-25% Feature freeze, focus on reliability
< 10% Emergency mode, all hands on stability

Budget Reset

Error budgets reset on the first day of each calendar month.

Alerting

SLO-Based Alerts

SLO Warning Critical
Accuracy < 99.95% (5min) < 99.9% (5min)
Latency p99 > 1500ms (5min) > 2000ms (5min)
Availability < 99.95% (1hr) < 99.9% (1hr)
Throughput < 800/s (5min) < 500/s (5min)

Alert Routing

  • Warning: #fizzbuzz-alerts Slack channel
  • Critical: PagerDuty → On-call engineer
  • Accuracy < 99.5%: Page SRE manager + AI team lead

Exclusions

The following are excluded from SLO calculations:

  1. Scheduled maintenance windows (announced 48h in advance)
  2. Upstream provider outages (Anthropic API, Upstash Kafka)
  3. Force majeure events
  4. Load testing traffic (tagged with x-load-test: true)

Reporting

Weekly SLO Review

Every Monday, the SRE team publishes:

  • SLO performance for the previous week
  • Error budget consumption
  • Incidents affecting SLOs
  • Trend analysis

Monthly SLO Report

First week of each month:

  • 30-day SLO summary
  • Error budget reset status
  • Capacity planning recommendations
  • SLO revision proposals (if any)

SLO Revision Process

SLOs are reviewed quarterly. To propose changes:

  1. Create an ADR documenting the change
  2. Provide data supporting the new target
  3. Get approval from:
    • SRE team lead
    • Product manager
    • Engineering manager
  4. Update this document
  5. Update alerting thresholds
  6. Communicate to stakeholders

Appendix: Calculation Details

Accuracy Measurement

Accuracy is measured by comparing LLM responses to mathematical truth:

def is_correct(number: int, result: str) -> bool:
    expected = fizzbuzz(number)  # Mathematical implementation
    return result == expected

def fizzbuzz(n: int) -> str:
    if n % 15 == 0:
        return "FizzBuzz"
    elif n % 3 == 0:
        return "Fizz"
    elif n % 5 == 0:
        return "Buzz"
    else:
        return str(n)

Latency Measurement Points

[Client] → [Ingestion] → [Kafka] → [Fizz/Buzz] → [Aggregator] → [API] → [Client]
    t0          t1          t2          t3            t4          t5        t6

e2e_latency = t6 - t0

Last updated: 2025-12-01 Next review: 2026-03-01 Owner: @fizzbuzz-sre-team