Skip to content

ADR-002-schema-registry.md

Latest commit

 

History

History
97 lines (62 loc) · 4.05 KB

ADR-002-schema-registry.md

File metadata and controls

97 lines (62 loc) · 4.05 KB
Raw

ADR-002: Schema Registry for Event-Driven Architecture

Status

Accepted

Date

2024-09-15

Context

Following the adoption of event-driven architecture (ADR-001), we must establish a governance model for the events flowing between services. As FizzBuzz Enterprise Edition scales to handle enterprise workloads, the contracts between producers and consumers become critical infrastructure.

Without formal schema governance, we risk:

  1. Schema Drift: Services evolving message formats independently, causing runtime deserialization failures
  2. Implicit Contracts: Event structures defined only in code, requiring source diving to understand
  3. Breaking Changes: No mechanism to detect incompatible changes before deployment
  4. Onboarding Friction: New engineers must reverse-engineer event formats from multiple codebases

Industry best practices mandate schema registry adoption for any production Kafka deployment. Per the Platform Engineering Standards (PES-2024), all event-driven systems must have documented, versioned schemas.

Decision

We will implement a formal Schema Registry using Apache Avro schemas from day one. Schemas will be registered with the Redpanda Schema Registry (bundled with our Kafka-compatible streaming platform).

Implementation

  1. Schema Definitions: All event schemas will be defined as Avro .avsc files in /schemas/avro/

  2. Schema Registration: Schemas will be registered with the Redpanda Schema Registry:

    • Local: http://localhost:18081
    • Production: Managed Redpanda Cloud endpoint

  3. Compatibility Mode: BACKWARD compatibility will be enforced, allowing:

    • New optional fields with defaults
    • Field deprecation (with default values)

    But preventing:

    • Required field removal
    • Type changes
    • Field renaming

  4. CI Validation: All PRs modifying schemas will:

    • Validate Avro syntax
    • Check compatibility against the registry

Schema Inventory

Topic Schema Description
numbers-topic NumberSubmitted Entry point: number submitted for processing
fizz-topic FizzResult Divisibility-by-3 determination with LLM metadata
buzz-topic BuzzResult Divisibility-by-5 determination with LLM metadata
fizzbuzz-topic FizzBuzzResult Aggregated final classification

Consequences

Positive

  • Contract Clarity: Event structures are explicitly documented and versioned from project inception
  • Evolution Safety: Incompatible changes are caught in CI before deployment
  • Discoverability: Schema Registry UI provides self-service documentation
  • Cross-Team Coordination: Schema change process formalizes producer-consumer communication
  • Compliance Ready: Meets PES-2024 requirements for event governance

Negative

  • Initial Overhead: Schemas must be defined before services can produce events
  • Build Complexity: Services must maintain type definitions matching Avro schemas
  • Learning Curve: Team members must learn Avro schema syntax

Neutral

  • Serialization Format: Initial implementation uses JSON serialization; Avro binary encoding available for future bandwidth optimization (~40% reduction)

Alternatives Considered

JSON Schema

Rejected: Less mature tooling for Kafka integration. Avro is the de facto standard for Kafka schema management with native Schema Registry support.

Protocol Buffers

Rejected: Would require additional infrastructure (protoc compilation) and doesn't integrate as seamlessly with Confluent/Redpanda Schema Registry. Avro's self-describing format better supports our polyglot environment (Go + TypeScript).

No Schema Registry

Rejected: Violates PES-2024 standards and creates unacceptable operational risk as the platform scales.

References