fix(router): scope hasToolResult to the current turn (multi-turn replay)#316
Merged
Conversation
…n replay) `match.hasToolResult` was a whole-conversation predicate: "does ANY message have role tool". In a multi-turn chat session that pins it to `true` forever after the first tool call, so a turn's leg-1 fixture (hasToolResult:false, the tool-call leg) can never match again — the 2nd+ user turn always carries an earlier turn's tool result. Result: every pill/tool after the first in a shared thread returns "No fixture matched" (reproduced in the CopilotKit showcase Beautiful Chat demo: pill 1 works, pill 2+ dead). Scope it to the CURRENT turn — messages AFTER the last user message. For a single-turn request this is identical to the old check, so existing fixtures are unaffected; multi-turn leg-1 fixtures now match again. Updated the unit tests to use realistic leg-2 shapes (tool result is the last message of the turn, not followed by a fresh user message) and added a multi-turn regression test.
commit: |
BenTaylorDev
approved these changes
Jul 20, 2026
jpr5
added a commit
that referenced
this pull request
Jul 20, 2026
…lt+text scoping (#319) ## Problem PR #316 (released as v1.37.3) scoped the replay-side `hasToolResult` check to the **current turn** (messages after the last `user` message) but left two paths inconsistent, so v1.37.3 can silently produce fixtures that never match their own recorded request. ## Root cause **1. Record/replay split-brain.** The matcher (`src/router.ts`) was changed to a turn-scoped check, but the recorder (`src/recorder.ts`, `buildFixtureMatch`) still STAMPED `hasToolResult` with a whole-conversation predicate `messages.some(m => m.role === "tool")`. Both operate on the same normalized `ChatCompletionRequest.messages`, so for a genuinely-recorded **turn-2 leg-1** request `[user, assistant, tool, user]` (a fresh question whose history still carries turn-1's tool result): - Recorder writes `hasToolResult: true` (whole conversation has a tool). - Matcher on replay computes `false` (nothing after the last user) → `true !== false` → the fixture can never match its own recorded request. The comment at `recorder.ts` even claimed the value was "matcher-aware" — it no longer was. **2. Anthropic `tool_result` + text ordering.** Anthropic bundles a leg-2 `tool_result` and accompanying user text into ONE user message. Normalization emitted `[..., tool, user(text)]`, which is **byte-identical** to a genuine leg-1 turn, so the turn-scoped check saw no tool after the last user and misclassified a real leg-2 turn as `false`. Because the two shapes are indistinguishable *after* normalization, this can only be fixed where the source-message grouping is still known — in normalization, not in the shared helper. ## Fix - Extracted one shared `currentTurnHasToolResult(messages)` helper in `src/router.ts` (where the other message-shape utilities live and which `recorder.ts` already imports) and used it on **both** the record side and the match side. Sharing a single predicate makes the two sides structurally incapable of drifting. Replaced the false "matcher-aware" comment. - Fixed `claudeToCompletionRequest` to emit the accompanying text user message **before** the tool message(s), so a real Anthropic leg-2-with-text turn normalizes to `[..., user(text), tool]` and is classified `true`, while a genuine leg-1 turn (separate messages) stays `false`. The common `tool_result`-only follow-up is unchanged. ## Red-green proof Both defects are proven on the **real HTTP record→replay surface** (spin up a mock upstream, record through the real recorder proxy to disk, replay the recorded fixtures through the real server/matcher). Source fixes were `git stash`-ed to observe RED, then restored for GREEN — tests unchanged between runs. ### RED (source fixes reverted) ``` FAIL hasToolResult record→replay symmetry > replays a genuinely-recorded turn-2 leg-1 request against its own recorded fixture AssertionError: expected 503 to be 200 // Object.is equality - Expected 200 + Received 503 FAIL hasToolResult record→replay symmetry > replays an Anthropic leg-2 turn that bundles tool_result WITH accompanying text AssertionError: expected 503 to be 200 // Object.is equality - Expected 200 + Received 503 FAIL claudeToCompletionRequest > handles tool_result with text blocks alongside in same user message AssertionError: expected 'tool' to be 'user' // Object.is equality Expected: "user" Received: "tool" ``` (503 = strict-mode "No fixture matched": the recorded fixture could not match its own request.) ### GREEN (source fixes restored, tests unchanged) ``` ✓ src/__tests__/hastoolresult-record-replay-symmetry.test.ts (2 tests) ✓ src/__tests__/router.test.ts (144 tests) ✓ src/__tests__/messages.test.ts (73 tests) ✓ src/__tests__/thinking-invariants.test.ts (47 tests) Test Files 4 passed (4) Tests 266 passed (266) ``` Both the OpenAI split-brain (Finding 1) and the Anthropic tool_result+text case (Finding 2) are covered by the real-path HTTP record→replay test, so both got a genuine RED→GREEN on the actual failure surface (not tests-against-fakes). ## Tests added - `src/__tests__/hastoolresult-record-replay-symmetry.test.ts` — full HTTP record→replay for (a) an OpenAI turn-2 leg-1 request and (b) an Anthropic `tool_result`+text leg-2 turn; both would 503 pre-fix. - `router.test.ts` — direct `currentTurnHasToolResult` coverage (leg-2 true; new-turn turn-scoped false; leading-system-message; `lastUserIdx === -1` no-user whole-conversation fallback), plus matcher composition of the scoping with explicit `turnIndex` and `sequenceIndex` gates, and a system-message-led new-turn case. - `messages.test.ts` — the two existing tool_result+text tests that encoded the old (buggy) tool-before-text order were corrected to the fixed order. ## Full suite `pnpm format:check`, `pnpm lint`, `pnpm build` all clean. `pnpm test`: 4683 passed, 1 pre-existing wall-clock flake (`timing-replay.test.ts > replaySpeed 2.0 halves the replay duration`, `expected 209 to be less than 200`) that passes in isolation and is unrelated to this change. ## Notes No version bump and no changeset (repo has no `.changeset/`; releases are cut by a separate release PR). This completes the incomplete fix from #316. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Merged
jpr5
added a commit
that referenced
this pull request
Jul 20, 2026
Release v1.37.4. Bumps `package.json` to publish the fix from #319 to npm + ghcr. ## What ships - **#319** — `hasToolResult` record/replay symmetry via shared `currentTurnHasToolResult` helper (recorder previously stamped whole-conversation while the matcher read current-turn, so genuinely-recorded multi-turn fixtures could never match), plus Anthropic `tool_result`+text ordering fix. Completes the incomplete #316 / v1.37.3 fix. - CHANGELOG entries for 1.37.4 and the retroactive 1.37.3 (which shipped undocumented) already landed in #319. Consumer: unblocks CopilotKit showcase multi-turn recording once bumped to 1.37.4. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
pull Bot
pushed a commit
to TheTechOddBug/aimock
that referenced
this pull request
Jul 20, 2026
…ol_result+text scoping PR CopilotKit#316 scoped the replay-side hasToolResult check to the current turn (messages after the last user message) but left the record side and one Anthropic normalization case inconsistent, so v1.37.3 can silently produce fixtures that never match their own recorded request. - Extract one shared currentTurnHasToolResult(messages) helper in router.ts and use it on BOTH the record side (recorder.ts buildFixtureMatch, which previously stamped a whole-conversation messages.some(role==="tool")) and the match side (router.ts matchFixtureDiagnostic). A genuinely-recorded turn-2 leg-1 request was stamped true (whole-conversation) but matched false (turn-scoped), so its fixture could never replay. Sharing one predicate makes drift impossible. Also replaces the now-false "matcher-aware" comment in recorder.ts. - Fix Anthropic tool_result-with-text normalization: a user message bundling a tool_result WITH text normalized to [..., tool, user(text)], byte-identical to a genuine leg-1 turn, so a real leg-2 turn was misclassified false. Emit the accompanying text BEFORE the tool message so the tool trails the last user message and the turn is correctly classified true. This is the only place the source-message grouping is known, so it cannot be fixed in the shared helper. Proven red->green on the real HTTP record->replay surface (see PR body).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
match.hasToolResultis a whole-conversation predicate — "does ANY message have roletool". In a multi-turn chat session that pins it totrueforever after the first tool call, so a turn's leg-1 fixture (hasToolResult: false, the tool-call leg) can never match again: every 2nd+ user turn carries an earlier turn's tool result.Symptom in the CopilotKit showcase (Beautiful Chat): the first pill/tool in a thread works, but every pill after it returns "No fixture matched" (schedule-meeting, excalidraw, calculator, etc.). Reproduced + verified fixed against the live showcase.
Fix
Scope
hasToolResultto the current turn — messages after the lastusermessage. For a single-turn request this is identical to the old check (so existing fixtures are unaffected); multi-turn leg-1 fixtures match again.Tests
Updated the
hasToolResultunit tests to use realistic leg-2 shapes (the tool result is the last message of the turn, not followed by a fresh user message — the old tests appended a trailing user message, which is actually the multi-turn leg-1 case) and added a multi-turn regression test. Full suite green (4448 tests).Consumer note
CopilotKit showcase needs the released version (npm + ghcr) to fix multi-turn manual testing; CI single-turn probes are unaffected either way.