fix(anthropic,gemini): preserve provider-specific reasoning signatures across turns#1153
fix(anthropic,gemini): preserve provider-specific reasoning signatures across turns#1153tbille wants to merge 3 commits into
Conversation
WalkthroughThis PR adds ChangesThought signature preservation
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
... and 36 files with indirect coverage changes 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/any_llm/providers/anthropic/utils.py`:
- Around line 201-212: The assistant message conversion in anthropic/utils.py
needs to handle `content=None` before building `content_blocks`, since
`_build_anthropic_thinking_block` can produce a thinking block even when
`message["content"]` is null. Update the `elif message["role"] == "assistant"`
branch to explicitly treat `None` like empty content in the `message` handling
logic, alongside the existing string and iterable cases, so the `content_blocks`
construction in the assistant conversion path never splats a null value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: bdbd21e0-5415-4f7b-a756-a90ed51e151f
📒 Files selected for processing (5)
src/any_llm/providers/anthropic/utils.pysrc/any_llm/providers/gemini/utils.pysrc/any_llm/types/completion.pytests/unit/providers/test_anthropic_provider.pytests/unit/providers/test_gemini_provider.py
|
Addressed the CodeRabbit review comment: guarded against Note: the |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/any_llm/providers/anthropic/utils.py (1)
60-66: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winTreat empty Anthropic signatures as absent.
Line 65 still returns
""as a valid signature, but Line 1025 in the new regression suite already defines the empty string as missing. That split means a caller can replayextra_content={"anthropic": {"signature": ""}}into athinkingblock with an emptysignature, which violates the Anthropic request contract on the next turn.Suggested fix
def _extract_anthropic_thinking_signature(message: dict[str, Any]) -> str | None: """Extract the encrypted thinking signature stored on a message's extra_content, if any.""" extra_content = message.get("extra_content") if isinstance(extra_content, dict) and isinstance(anthropic_extra := extra_content.get("anthropic"), dict): signature = anthropic_extra.get("signature") - if isinstance(signature, str): + if isinstance(signature, str) and signature: return signature return None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/any_llm/providers/anthropic/utils.py` around lines 60 - 66, The helper _extract_anthropic_thinking_signature currently treats an empty string as a valid Anthropic signature. Update this function so only non-empty string signatures are returned, and return None for empty values from extra_content["anthropic"]["signature"]. This keeps the signature handling consistent with the regression suite and prevents replaying an empty signature into the next thinking block.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/any_llm/providers/anthropic/utils.py`:
- Around line 60-66: The helper _extract_anthropic_thinking_signature currently
treats an empty string as a valid Anthropic signature. Update this function so
only non-empty string signatures are returned, and return None for empty values
from extra_content["anthropic"]["signature"]. This keeps the signature handling
consistent with the regression suite and prevents replaying an empty signature
into the next thinking block.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ddc89df8-bfef-4f52-96d3-8f2dc429ec94
📒 Files selected for processing (2)
src/any_llm/providers/anthropic/utils.pytests/unit/providers/test_anthropic_provider.py
…s across turns Both Gemini and Anthropic encrypt their internal reasoning trace and expose it as a signature that must be replayed unmodified on subsequent turns (Gemini's Part.thought_signature, Anthropic's ThinkingBlock/SignatureDelta signature). Neither value was being captured in any-llm's streaming conversion, and Anthropic's signature wasn't captured at all (streaming or non-streaming), nor replayed on outgoing requests, which can lead to degraded or repeated reasoning in multi-turn tool-use conversations. - Add extra_content to ChatCompletionMessage and ChoiceDelta (mirroring the existing tool-call-level extra_content) to carry provider-specific, message-level metadata such as Anthropic's thinking signature. - Add a ChoiceDeltaToolCall subclass with extra_content so streaming tool call deltas can carry the same per-call metadata as the non-streaming ChatCompletionMessageFunctionToolCall (e.g. Gemini's thought_signature). - Gemini: capture thought_signature on streaming function-call deltas, matching the existing non-streaming behavior. - Anthropic: capture the thinking block signature in both streaming (signature_delta) and non-streaming responses, and replay it as an Anthropic 'thinking' content block when continuing a conversation that used extended thinking (with or without an accompanying tool call). Fixes #1149
…king blocks CodeRabbit flagged that a reasoning-only assistant turn (content=None, no tool_calls) would crash with a TypeError when replaying the thinking block, since the code assumed any non-string content was an iterable list. Handle None explicitly and only extend content_blocks for actual list content. Also adds coverage for the remaining new branches: non-string signatures, missing/malformed reasoning text, list-shaped existing content, empty-string content, and an empty (falsy) thinking signature in the non-streaming path.
tbille
force-pushed
the
fix/preserve-reasoning-signatures-1149
branch
from
4f77d67 to
ea606fc
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/any_llm/providers/gemini/utils.py`:
- Around line 408-414: Extract the duplicated thought_signature-to-extra_content
encoding logic into a shared helper in the Gemini utils module, and update both
the streaming path and _convert_response_to_response_dict to call it. Keep the
helper responsible for checking thought_signature, base64-encoding bytes, and
returning the google extra_content payload so both call sites stay consistent
and avoid drift.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: af99abc5-836c-469a-ba95-e47fe640ed1c
📒 Files selected for processing (5)
src/any_llm/providers/anthropic/utils.pysrc/any_llm/providers/gemini/utils.pysrc/any_llm/types/completion.pytests/unit/providers/test_anthropic_provider.pytests/unit/providers/test_gemini_provider.py
CodeRabbit nitpick: the thought_signature-to-extra_content encoding logic was duplicated identically between the non-streaming (_convert_response_to_response_dict) and streaming (_create_openai_chunk_from_google_chunk) conversion paths. Extract it into a shared helper so the two paths can't drift.
|
Addressed CodeRabbit's latest nitpick: extracted a shared Also: the branch has since been rebased onto |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/any_llm/providers/gemini/utils.py`:
- Around line 279-288: The helper `_thought_signature_extra_content` is using a
generic `Any` lookup for `thought_signature` even though it only handles
`google.genai.types.Part` objects. Update the function signature to use
`types.Part` and read `part.thought_signature` directly, keeping the existing
bytes check and base64 payload shape so both response conversion paths remain
consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: c5220628-552e-44fe-99de-7b42475ccd1a
📒 Files selected for processing (1)
src/any_llm/providers/gemini/utils.py
| def _thought_signature_extra_content(part: Any) -> dict[str, Any] | None: | ||
| """Build the OpenAI-compatible extra_content payload for a Gemini thought_signature, if present. | ||
|
|
||
| Shared by both the non-streaming (_convert_response_to_response_dict) and streaming | ||
| (_create_openai_chunk_from_google_chunk) conversion paths so they stay in sync. | ||
| """ | ||
| thought_signature = getattr(part, "thought_signature", None) | ||
| if thought_signature is not None and isinstance(thought_signature, bytes): | ||
| return {"google": {"thought_signature": base64.b64encode(thought_signature).decode("utf-8")}} | ||
| return None |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file around the reported lines.
sed -n '240,320p' src/any_llm/providers/gemini/utils.py
# Find where types.Part is imported and used in this file.
rg -n "types\.Part|from .* import .*types|import .*types" src/any_llm/providers/gemini/utils.py
# Locate the definition of Part in the repository, if present.
fd -a "types.py" src any_llm . | sed -n '1,40p'
rg -n "class Part|def .*thought_signature|thought_signature" src any_llm . -g '!**/.git/**'Repository: mozilla-ai/any-llm
Length of output: 11963
🌐 Web query:
google genai types.Part thought_signature attribute documented Part thought_signature bytes
💡 Result:
In the Google Gen AI ecosystem, thought_signature is a field used to maintain reasoning continuity and state for models that support thinking [1][2]. It acts as an opaque, encrypted representation of the model's internal reasoning process [2][3]. Regarding the Part object and the thought_signature data type: 1. Data Type: The thought_signature attribute is generally handled as a string [4][5]. While some SDKs (notably Go) may define the type internally as []byte [6], it is functionally an opaque string provided by the API [4]. Developers are advised that if an SDK uses []byte, standard JSON marshaling might incorrectly encode the value (e.g., base64 encoding a bypass string), which can cause the signature to become non-functional [6]. The Python and JavaScript SDKs typically handle this as a string, allowing for straightforward pass-through of the value [4][6][5]. 2. Purpose and Usage: - Preservation: It is used to preserve the Gemini reasoning state in multi-turn and multi-step conversations, particularly when function calling is involved [1][2]. - Requirements: You must include all Part objects from previous messages exactly as they were returned by the model [2][3]. You should not merge, modify, or omit parts that contain these signatures, as this can degrade reasoning quality or trigger 400 errors in strict validation scenarios (such as with the first functionCall part in a turn) [1][2]. - Non-Function Call Parts: Gemini may also return thought_signatures in the final Part of a response (e.g., text parts) even when no function call occurs [1][2]. While returning these is not strictly enforced with a blocking error, it is recommended to ensure high-quality reasoning and performance [1][2]. 3. Documentation: The field is documented as an optional attribute within the Part interface (e.g., Part.thoughtSignature) [4][5]. It is explicitly described as an opaque signature designed to be reused in subsequent requests to maintain context [4][7][5].
Citations:
- 1: https://ai.google.dev/gemini-api/docs/generate-content/thought-signatures
- 2: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/thought-signatures
- 3: https://ai.google.dev/gemini-api/docs/thought-signatures
- 4: https://googleapis.github.io/js-genai/release_docs/interfaces/types.Part.html
- 5: googleapis/js-genai@20815b2
- 6: Part.ThoughtSignature bypass string is base64-encoded by standard JSON marshaling, making it non-functional googleapis/go-genai#711
- 7: googleapis/python-genai@303f906
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '80,150p' src/any_llm/types/completion.py
sed -n '860,930p' tests/unit/providers/test_gemini_provider.py
sed -n '980,1065p' tests/unit/providers/test_gemini_provider.pyRepository: mozilla-ai/any-llm
Length of output: 8962
🌐 Web query:
python-genai Part thought_signature attribute type bytes docs
💡 Result:
In the Google Gen AI Python SDK (google-genai), the thought_signature attribute of the Part class is defined as bytes [1][2]. Key details regarding this attribute include: - Purpose: It is an opaque, encrypted representation of the model's internal reasoning (thought) process [3][4][5]. It is required to maintain reasoning continuity across multi-turn interactions, particularly when using thinking-enabled models [3][6][7]. - API Behavior: When Gemini generates content (such as function calls or text), it may include a thought_signature in the Part [3][6]. If the model response contains a functionCall, including the thought_signature when sending the conversation history back is mandatory; omitting it often results in a 400 error [3][6][8]. - SDK Implementation: The SDK models this field within the Part class as Optional[bytes] [1][2]. While the underlying REST API often represents this signature as a string (frequently base64-encoded) [9][5], the Python SDK types it as bytes [1][2]. - Handling: When using standard chat history features or appending the full model response to the history, the official SDK typically handles the attachment of the thought_signature automatically behind the scenes [6]. However, users have reported issues where the signature may not be correctly passed back in certain scenarios (such as specific streaming patterns or multi-turn function calls), requiring careful management of the content history [10][8]. Documentation on this field can be found in the SDK source code types definitions [11][2] and the Gemini API documentation regarding thinking models [3][6][4].
Citations:
- 1: googleapis/python-genai@303f906
- 2: https://github.com/googleapis/python-genai/blob/6196b1b4251007e33661bb5d7dc27bafee3feefe/google/genai/types.py
- 3: https://ai.google.dev/gemini-api/docs/generate-content/thought-signatures
- 4: https://ai.google.dev/gemini-api/docs/thought-signatures
- 5: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/thinking/intro_thought_signatures_rest.ipynb
- 6: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/thought-signatures
- 7: https://ai.google.dev/gemini-api/docs/interactions/thinking
- 8: AsyncChat.send_message_stream does not pass thought_signature with function calls googleapis/python-genai#1938
- 9: googleapis/js-genai@20815b2
- 10:
thought_signatureinvalid when pass back to sdk googleapis/python-genai#2105 - 11: https://github.com/googleapis/python-genai/blob/4b5720a7/google/genai/types.py
Use types.Part here and access thought_signature directly src/any_llm/providers/gemini/utils.py:279
The helper only receives google.genai.types.Part objects, and thought_signature is a declared field there, so part: types.Part plus part.thought_signature is the cleaner type-safe form.
🧰 Tools
🪛 Ruff (0.15.20)
[warning] 279-279: Dynamically typed expressions (typing.Any) are disallowed in part
(ANN401)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/any_llm/providers/gemini/utils.py` around lines 279 - 288, The helper
`_thought_signature_extra_content` is using a generic `Any` lookup for
`thought_signature` even though it only handles `google.genai.types.Part`
objects. Update the function signature to use `types.Part` and read
`part.thought_signature` directly, keeping the existing bytes check and base64
payload shape so both response conversion paths remain consistent.
Sources: Coding guidelines, Linters/SAST tools
1 participant
Description
Both Gemini and Anthropic encrypt their internal extended-reasoning trace and
expose it via a "signature" that the API requires to be replayed unmodified
on subsequent turns:
Part.thought_signatureThinkingBlock.signature/SignatureDelta.signature(streaming)Neither value was being captured in any-llm's streaming conversion path.
Anthropic's signature additionally was not captured at all in the
non-streaming path, and was never replayed on outgoing requests, so any
multi-turn conversation using extended thinking + tool calls lost the
original reasoning trace on the next turn (matching the "weird thinking
loops" symptom reported in the issue).
Changes
any_llm/types/completion.py: addextra_contenttoChatCompletionMessageand
ChoiceDelta(mirroring the existing tool-call-levelextra_content)to carry provider-specific, message-level metadata. Add a
ChoiceDeltaToolCallsubclass withextra_contentso streaming tool-calldeltas can carry the same per-call metadata as the non-streaming
ChatCompletionMessageFunctionToolCall(e.g. Gemini'sthought_signature).gemini/utils.py: capturethought_signatureon streaming function-calldeltas, matching the existing non-streaming behavior.
anthropic/utils.py:signaturein both streaming(
signature_deltaevent) and non-streaming responses.thinkingcontent block when continuing aconversation that used extended thinking, both alongside a tool call and
in plain assistant turns, per Anthropic's docs on preserving thinking
blocks.
PR Type
Relevant issues
Fixes #1149
Checklist
AI Usage Information
AI Model used: Claude Sonnet 5
AI Developer Tool used: OpenCode
Any other info you'd like to share: Implemented end-to-end (analysis, fix, and tests) by an AI agent based on the linked issue.
I am an AI Agent filling out this form (check box if true)
Summary by CodeRabbit
extra_contentfield so this metadata can reach the client consistently.