Skip to content

fix(drift): classify full live /models family wave + durable preview-exclude rule#313

Merged
jpr5 merged 2 commits into
mainfrom
fix/drift-model-family-classification
Jul 18, 2026
Merged

fix(drift): classify full live /models family wave + durable preview-exclude rule#313
jpr5 merged 2 commits into
mainfrom
fix/drift-model-family-classification

Conversation

@jpr5

@jpr5 jpr5 commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Problem

The daily model-family drift canary (src/__tests__/drift/models.drift.ts)
flagged the entire 2026-07-16 live GET /models wave as UNCLASSIFIED
families present in neither includeFamilies nor excludeFamilies, which turns
the drift job red:

  • OpenAI: 48 unclassified families (gpt-5.x tiers, Codex line, o-series,
    retired base models, image/video/moderation/search/deep-research surfaces)
  • Anthropic: 10 (all Claude 4.x/5 point releases + claude-fable-5)
  • Gemini: 45 (stable flash/pro tiers, plus preview/image/audio/tts/video/
    music/robotics/embeddings/Gemma surfaces)

Root cause of the recurrence: excludeFamilies is a hand-maintained enumerated
list with no rule for the churn-heavy families (previews, Gemma variants),
so each new one re-fires the canary as brand-new drift and needs a manual
registry edit — forever.

Fix

Classification (per resolved decisions):

  • INCLUDE (text chat / reasoning aimock mocks): OpenAI gpt-5.x tiers + chat
    surfaces, the Codex line (gpt-5-codexgpt-5.3-codex),
    gpt-5.6-luna/-sol/-terra, o-series; Anthropic all 4.x/5 point releases
    including claude-fable-5 (included ahead of a recorded fixture — intended,
    mirrors the forward-looking rationale of the exclude-by-rule patterns); stable
    Gemini flash/pro tiers.
  • EXCLUDE: retired base (babbage/davinci/gpt-3.5-), embeddings, image
    (gpt-image
    /Imagen/nano-banana/gemini-*-image), audio/TTS, moderation, video
    (Sora/Veo), music (Lyria), robotics, deep-research, search, computer-use,
    antigravity, aqa.
  • Moving aliases (chat-latest, gemini *-latest): excluded as non-family
    aliases — no existing include convention for them.

Durable hardening — two exclude-by-rule predicates:

  • PREVIEW_FAMILY (/-preview(-\d+)?$/) — any family whose normalized key
    ends in -preview (optionally -preview-NN) is excluded by RULE.
    Previews are blanket-excluded until GA per policy: this intentionally
    silences the canary on new text previews (e.g. o1-preview, gpt-4.5-preview)
    until they drop the -preview suffix and go GA. There is no include-side
    escape hatch
    — aimock does not mock preview surfaces.
  • GEMMA_FAMILY (/^gemma(-|$)/) — the open-weight Gemma line is out of
    scope (aimock does not mock it; it only rides the shared Gemini /models
    list). Replaces the two enumerated literals (gemma-4-26b-a4b-it,
    gemma-4-31b-it) so future Gemma variants auto-exclude with zero registry
    churn
    . Reversible by explicit handling if a Gemma fixture is ever added.

Removed the dead include-wins escape hatch: the prior isClassifiedFamily
had an "include wins" branch so an included family ending in -preview would
still be treated as included. No consumer distinguished include from exclude for
preview families, so the branch was dead code, and the two tests guarding it
were false-green (they used gemini-x-chat-preview, which matches
PREVIEW_FAMILY, so they passed even with the branch deleted). Both the dead
branch and the false-green tests are removed. Verified
include ∩ (preview ∪ gemma) = ∅, so removal is a functional no-op.

A family is now classified iff it is in include ∪ exclude, or matches an
exclude-by-rule pattern (preview, gemma). Implemented at the classification
layer, not the normalizer
, so preview/gemma family keys stay distinct and the
canary keeps surfacing genuinely-new non-preview, non-gemma families. The
builder/fixture cross-check uses the same isClassifiedFamily predicate so the
two classification surfaces cannot drift apart.

No version bumppackage.json untouched (versioned separately).

Red → Green proof

include ∩ (preview ∪ gemma) = ∅ — verified empirically before removing the
include-wins branch: no family in any includeFamilies set matches
PREVIEW_FAMILY or GEMMA_FAMILY, so the removal cannot mis-classify any
included family.

GEMMA_FAMILY rule (new genuine red-green test):

RED — against the pre-fix registry (literal names only, no GEMMA_FAMILY):

 × isClassifiedFamily / PREVIEW_FAMILY > classifies a future Gemma variant by rule
   → Cannot read properties of undefined (reading 'test')   # GEMMA_FAMILY did not exist

A synthetic gemma-9-foo (normalizes to itself — no numeric-only build tag to
strip) was on neither include nor exclude and was unclassified under the old
literal-names-only set.

GREEN — with the pattern:

 ✓ isClassifiedFamily / PREVIEW_FAMILY > classifies a future Gemma variant by rule
 ✓ gemma families are excluded by rule > a future Gemma variant auto-excludes with zero registry edits

Full drift + unit suite (after the fix):

 Test Files  16 passed | 8 skipped (24)     # pnpm test:drift — 0/0/0 unclassified
      Tests  98 passed | 55 skipped (153)
 Test Files  158 passed (158)               # pnpm test — full unit
      Tests  4670 passed (4670)

The full-wave regression suites (OpenAI/Anthropic/Gemini) still assert zero
unclassified
against the real unclassifiedFamilies pipeline. The two
false-green INCLUDE-WINS tests are gone; no tautological test replaces them.

format:check / lint / tsdown build / commitlint (header ≤100) all pass on
the changed files.

…preview rule

The daily drift canary flagged the entire 2026-07-16 live /models wave as
UNCLASSIFIED: OpenAI 48, Anthropic 10, Gemini 45 new families beyond
include ∪ exclude, turning the job red.

Classify every family per the resolved decisions:

- INCLUDE (text chat / reasoning aimock mocks): OpenAI gpt-5.x tiers +
  chat surfaces, the Codex line (gpt-5-codex … gpt-5.3-codex),
  gpt-5.6-luna/-sol/-terra, o-series; Anthropic all 4.x/5 point releases
  incl. claude-fable-5; stable Gemini flash/pro tiers.
- EXCLUDE (retired / embeddings / image / audio / tts / moderation /
  video / music / robotics / deep-research / search / computer-use /
  antigravity / aqa, and the open-weight Gemma line which aimock does
  not mock). Moving aliases (chat-latest, gemini *-latest) excluded as
  non-family aliases per the existing convention.

Durable hardening — a registry-level PREVIEW_FAMILY predicate
(/-preview(-\d+)?$/) treats any trailing-preview family as
excluded-by-RULE, with INCLUDE WINS (a family explicitly on
includeFamilies is still included even if it ends in -preview). This
auto-excludes the 5 Gemini-3.x previews and every future preview tier
with zero registry churn — the enumeration approach re-alerted on each
new preview wave. Implemented at the classification layer, not the
normalizer, so preview family keys stay distinct and the canary keeps
surfacing genuinely-new non-preview families. Interior -preview-<word>
suffixes (-preview-tts, -preview-customtools) are enumerated explicitly
so their category stays documented. The builder/fixture cross-check now
uses the same isClassifiedFamily predicate so both classification
surfaces cannot drift apart.

Red/green: the new full-wave regression tests inject the exact flagged
families into the real unclassifiedFamilies pipeline (48/10/45 before →
0/0/0 after); preview-predicate unit tests cover auto-exclude,
include-wins, and interior-suffix non-match.
@pkg-pr-new

pkg-pr-new Bot commented Jul 18, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@313

commit: 043410b

…ape hatch

The two "INCLUDE WINS" tests were false-green: they used
gemini-x-chat-preview, which matches PREVIEW_FAMILY, so they passed even
with the include-wins branch deleted. The include-wins branch in
isClassifiedFamily was dead — no consumer distinguished include from
exclude for preview families.

Per policy, previews are blanket-excluded until GA (canary intentionally
silent on new text previews until they drop the -preview suffix), so the
escape hatch has no purpose. Changes:

- Remove the dead include-wins branch from isClassifiedFamily. A family
  is now classified iff it is in include ∪ exclude, or matches an
  exclude-by-rule pattern. Verified include ∩ (preview ∪ gemma) = ∅, so
  the removal is a functional no-op.
- Remove the two false-green INCLUDE-WINS tests (registry + drift) rather
  than leave tautological assertions behind.
- Upgrade gemma to a GEMMA_FAMILY exclude-by-rule pattern (/^gemma(-|$)/)
  replacing the two literal names, so future gemma variants auto-exclude
  with zero registry churn. Add a genuine red-green test: gemma-9-foo
  classifies as excluded-by-rule.
- Document the preview tradeoff near PREVIEW_FAMILY and fix the misleading
  "2-digit" comment the /-preview(-\d+)?$/ regex does not enforce.
- Fix the 48/10/45 unclassified-count comment (was 49/10/44) and note
  claude-fable-5 is included ahead of a recorded fixture.
@jpr5
jpr5 marked this pull request as ready for review July 18, 2026 17:32
@jpr5
jpr5 merged commit 761bd3a into main Jul 18, 2026
27 checks passed
@jpr5
jpr5 deleted the fix/drift-model-family-classification branch July 18, 2026 17:32
jpr5 added a commit that referenced this pull request Jul 18, 2026
…#315)

## Summary

Fixes the live drift job residual from run 29654128652 (2026-07-18,
main): bare `chat-latest` was returned by OpenAI's `/models` but not
present in either `includeFamilies` or `excludeFamilies` in
`model-registry.ts`, causing a critical unclassified-family signal.

**Root cause:** `chat-latest` is a bare moving alias with no per-tier
prefix (unlike `gpt-5-chat-latest`, `gpt-5.1/5.2/5.3-chat-latest`). The
normalizer leaves it unchanged (no date/build-tag suffix to strip), so
it fell through as an unclassified family.

**Fix:** Added `chat-latest` to `excludeFamilies.openai` — consistent
with PR #313's stated policy that moving aliases belong in
excludeFamilies (same rationale as `chatgpt-image-latest`,
`omni-moderation-latest`, `gemini-flash-latest`, etc.).

**Fixture-vs-live gap closed:** The static OpenAI wave in
`models.drift.ts` previously omitted bare `chat-latest`, so the test
suite ran 0/0/0 for this family while the live `/models` endpoint
returned it. Added `chat-latest` to the wave so the static suite will
exercise this family class going forward.

**Sibling scan:** Audited all `*-latest` entries across
openai/gemini/anthropic. Every other bare alias is already classified
(include or exclude). `chat-latest` was the only unclassified one.

## Red-green proof

**RED** (wave has `chat-latest`, registry does not):
```
× src/__tests__/drift/models.drift.ts > full live /models wave is fully classified
  → expected [ 'chat-latest' ] to deeply equal []
```
Unclassified count: 1

**GREEN** (after adding to excludeFamilies):
```
✓ src/__tests__/drift/models.drift.ts > full live /models wave is fully classified
✓ ...OpenAI: every live family is classified (zero unclassified)
Test Files: 158 passed (158) — Tests: 4673 passed (4673)
```
Unclassified count: 0

## No version bump

`git diff origin/main -- package.json` → empty (zero diff).

## Changes

- `src/__tests__/drift/model-registry.ts` — added `chat-latest` to
`excludeFamilies.openai` with comment
- `src/__tests__/drift/models.drift.ts` — added `chat-latest` to the
static OpenAI wave (fixture-vs-live gap fix)
pull Bot pushed a commit to TheTechOddBug/aimock that referenced this pull request Jul 18, 2026
chat-latest is a moving alias on OpenAI's /models endpoint with no
per-tier prefix (unlike gpt-5-chat-latest, gpt-5.1-chat-latest, etc.).
The normalizer leaves it unchanged (no date/build-tag suffix to strip),
so it fell through the registry as an unclassified family. Per PR CopilotKit#313's
stated policy, moving aliases belong in excludeFamilies.

Also adds chat-latest to the static OpenAI wave in models.drift.ts to
close the fixture-vs-live gap: the static suite previously omitted it,
so the test exercised 0/0/0 while the live /models returned it (making
this class of bare alias invisible to the test).

Verified sibling *-latest entries across openai/gemini/anthropic — all
other bare aliases already appear in include or excludeFamilies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant