fix(drift): classify full live /models family wave + durable preview-exclude rule#313
Merged
Merged
Conversation
…preview rule The daily drift canary flagged the entire 2026-07-16 live /models wave as UNCLASSIFIED: OpenAI 48, Anthropic 10, Gemini 45 new families beyond include ∪ exclude, turning the job red. Classify every family per the resolved decisions: - INCLUDE (text chat / reasoning aimock mocks): OpenAI gpt-5.x tiers + chat surfaces, the Codex line (gpt-5-codex … gpt-5.3-codex), gpt-5.6-luna/-sol/-terra, o-series; Anthropic all 4.x/5 point releases incl. claude-fable-5; stable Gemini flash/pro tiers. - EXCLUDE (retired / embeddings / image / audio / tts / moderation / video / music / robotics / deep-research / search / computer-use / antigravity / aqa, and the open-weight Gemma line which aimock does not mock). Moving aliases (chat-latest, gemini *-latest) excluded as non-family aliases per the existing convention. Durable hardening — a registry-level PREVIEW_FAMILY predicate (/-preview(-\d+)?$/) treats any trailing-preview family as excluded-by-RULE, with INCLUDE WINS (a family explicitly on includeFamilies is still included even if it ends in -preview). This auto-excludes the 5 Gemini-3.x previews and every future preview tier with zero registry churn — the enumeration approach re-alerted on each new preview wave. Implemented at the classification layer, not the normalizer, so preview family keys stay distinct and the canary keeps surfacing genuinely-new non-preview families. Interior -preview-<word> suffixes (-preview-tts, -preview-customtools) are enumerated explicitly so their category stays documented. The builder/fixture cross-check now uses the same isClassifiedFamily predicate so both classification surfaces cannot drift apart. Red/green: the new full-wave regression tests inject the exact flagged families into the real unclassifiedFamilies pipeline (48/10/45 before → 0/0/0 after); preview-predicate unit tests cover auto-exclude, include-wins, and interior-suffix non-match.
commit: |
…ape hatch The two "INCLUDE WINS" tests were false-green: they used gemini-x-chat-preview, which matches PREVIEW_FAMILY, so they passed even with the include-wins branch deleted. The include-wins branch in isClassifiedFamily was dead — no consumer distinguished include from exclude for preview families. Per policy, previews are blanket-excluded until GA (canary intentionally silent on new text previews until they drop the -preview suffix), so the escape hatch has no purpose. Changes: - Remove the dead include-wins branch from isClassifiedFamily. A family is now classified iff it is in include ∪ exclude, or matches an exclude-by-rule pattern. Verified include ∩ (preview ∪ gemma) = ∅, so the removal is a functional no-op. - Remove the two false-green INCLUDE-WINS tests (registry + drift) rather than leave tautological assertions behind. - Upgrade gemma to a GEMMA_FAMILY exclude-by-rule pattern (/^gemma(-|$)/) replacing the two literal names, so future gemma variants auto-exclude with zero registry churn. Add a genuine red-green test: gemma-9-foo classifies as excluded-by-rule. - Document the preview tradeoff near PREVIEW_FAMILY and fix the misleading "2-digit" comment the /-preview(-\d+)?$/ regex does not enforce. - Fix the 48/10/45 unclassified-count comment (was 49/10/44) and note claude-fable-5 is included ahead of a recorded fixture.
jpr5
marked this pull request as ready for review
July 18, 2026 17:32
jpr5
added a commit
that referenced
this pull request
Jul 18, 2026
…#315) ## Summary Fixes the live drift job residual from run 29654128652 (2026-07-18, main): bare `chat-latest` was returned by OpenAI's `/models` but not present in either `includeFamilies` or `excludeFamilies` in `model-registry.ts`, causing a critical unclassified-family signal. **Root cause:** `chat-latest` is a bare moving alias with no per-tier prefix (unlike `gpt-5-chat-latest`, `gpt-5.1/5.2/5.3-chat-latest`). The normalizer leaves it unchanged (no date/build-tag suffix to strip), so it fell through as an unclassified family. **Fix:** Added `chat-latest` to `excludeFamilies.openai` — consistent with PR #313's stated policy that moving aliases belong in excludeFamilies (same rationale as `chatgpt-image-latest`, `omni-moderation-latest`, `gemini-flash-latest`, etc.). **Fixture-vs-live gap closed:** The static OpenAI wave in `models.drift.ts` previously omitted bare `chat-latest`, so the test suite ran 0/0/0 for this family while the live `/models` endpoint returned it. Added `chat-latest` to the wave so the static suite will exercise this family class going forward. **Sibling scan:** Audited all `*-latest` entries across openai/gemini/anthropic. Every other bare alias is already classified (include or exclude). `chat-latest` was the only unclassified one. ## Red-green proof **RED** (wave has `chat-latest`, registry does not): ``` × src/__tests__/drift/models.drift.ts > full live /models wave is fully classified → expected [ 'chat-latest' ] to deeply equal [] ``` Unclassified count: 1 **GREEN** (after adding to excludeFamilies): ``` ✓ src/__tests__/drift/models.drift.ts > full live /models wave is fully classified ✓ ...OpenAI: every live family is classified (zero unclassified) Test Files: 158 passed (158) — Tests: 4673 passed (4673) ``` Unclassified count: 0 ## No version bump `git diff origin/main -- package.json` → empty (zero diff). ## Changes - `src/__tests__/drift/model-registry.ts` — added `chat-latest` to `excludeFamilies.openai` with comment - `src/__tests__/drift/models.drift.ts` — added `chat-latest` to the static OpenAI wave (fixture-vs-live gap fix)
pull Bot
pushed a commit
to TheTechOddBug/aimock
that referenced
this pull request
Jul 18, 2026
chat-latest is a moving alias on OpenAI's /models endpoint with no per-tier prefix (unlike gpt-5-chat-latest, gpt-5.1-chat-latest, etc.). The normalizer leaves it unchanged (no date/build-tag suffix to strip), so it fell through the registry as an unclassified family. Per PR CopilotKit#313's stated policy, moving aliases belong in excludeFamilies. Also adds chat-latest to the static OpenAI wave in models.drift.ts to close the fixture-vs-live gap: the static suite previously omitted it, so the test exercised 0/0/0 while the live /models returned it (making this class of bare alias invisible to the test). Verified sibling *-latest entries across openai/gemini/anthropic — all other bare aliases already appear in include or excludeFamilies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The daily model-family drift canary (
src/__tests__/drift/models.drift.ts)flagged the entire 2026-07-16 live
GET /modelswave as UNCLASSIFIED —families present in neither
includeFamiliesnorexcludeFamilies, which turnsthe drift job red:
retired base models, image/video/moderation/search/deep-research surfaces)
claude-fable-5)music/robotics/embeddings/Gemma surfaces)
Root cause of the recurrence:
excludeFamiliesis a hand-maintained enumeratedlist with no rule for the churn-heavy families (previews, Gemma variants),
so each new one re-fires the canary as brand-new drift and needs a manual
registry edit — forever.
Fix
Classification (per resolved decisions):
surfaces, the Codex line (
gpt-5-codex…gpt-5.3-codex),gpt-5.6-luna/-sol/-terra, o-series; Anthropic all 4.x/5 point releasesincluding
claude-fable-5(included ahead of a recorded fixture — intended,mirrors the forward-looking rationale of the exclude-by-rule patterns); stable
Gemini flash/pro tiers.
(gpt-image/Imagen/nano-banana/gemini-*-image), audio/TTS, moderation, video
(Sora/Veo), music (Lyria), robotics, deep-research, search, computer-use,
antigravity, aqa.
chat-latest, gemini*-latest): excluded as non-familyaliases — no existing include convention for them.
Durable hardening — two exclude-by-rule predicates:
PREVIEW_FAMILY(/-preview(-\d+)?$/) — any family whose normalized keyends in
-preview(optionally-preview-NN) is excluded by RULE.Previews are blanket-excluded until GA per policy: this intentionally
silences the canary on new text previews (e.g.
o1-preview,gpt-4.5-preview)until they drop the
-previewsuffix and go GA. There is no include-sideescape hatch — aimock does not mock preview surfaces.
GEMMA_FAMILY(/^gemma(-|$)/) — the open-weight Gemma line is out ofscope (aimock does not mock it; it only rides the shared Gemini
/modelslist). Replaces the two enumerated literals (
gemma-4-26b-a4b-it,gemma-4-31b-it) so future Gemma variants auto-exclude with zero registrychurn. Reversible by explicit handling if a Gemma fixture is ever added.
Removed the dead include-wins escape hatch: the prior
isClassifiedFamilyhad an "include wins" branch so an included family ending in
-previewwouldstill be treated as included. No consumer distinguished include from exclude for
preview families, so the branch was dead code, and the two tests guarding it
were false-green (they used
gemini-x-chat-preview, which matchesPREVIEW_FAMILY, so they passed even with the branch deleted). Both the deadbranch and the false-green tests are removed. Verified
include ∩ (preview ∪ gemma) = ∅, so removal is a functional no-op.A family is now classified iff it is in
include ∪ exclude, or matches anexclude-by-rule pattern (preview, gemma). Implemented at the classification
layer, not the normalizer, so preview/gemma family keys stay distinct and the
canary keeps surfacing genuinely-new non-preview, non-gemma families. The
builder/fixture cross-check uses the same
isClassifiedFamilypredicate so thetwo classification surfaces cannot drift apart.
No version bump —
package.jsonuntouched (versioned separately).Red → Green proof
include ∩ (preview ∪ gemma) = ∅— verified empirically before removing theinclude-wins branch: no family in any
includeFamiliesset matchesPREVIEW_FAMILYorGEMMA_FAMILY, so the removal cannot mis-classify anyincluded family.
GEMMA_FAMILY rule (new genuine red-green test):
RED — against the pre-fix registry (literal names only, no
GEMMA_FAMILY):A synthetic
gemma-9-foo(normalizes to itself — no numeric-only build tag tostrip) was on neither include nor exclude and was unclassified under the old
literal-names-only set.
GREEN — with the pattern:
Full drift + unit suite (after the fix):
The full-wave regression suites (OpenAI/Anthropic/Gemini) still assert zero
unclassified against the real
unclassifiedFamiliespipeline. The twofalse-green INCLUDE-WINS tests are gone; no tautological test replaces them.
format:check/lint/tsdown build/commitlint(header ≤100) all pass onthe changed files.