Enable streaming when response_format is set for gemini, openai, and …#1162
Enable streaming when response_format is set for gemini, openai, and …#1162liukidar wants to merge 1 commit into
Conversation
…anthropic completion endpoints.
WalkthroughRemoves prior restrictions that raised errors when ChangesStreaming with response_format support
Related PRs: None identified. Suggested labels: bug, enhancement, tests Suggested reviewers: None identified. Poem 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Enables streaming chat completions when response_format is configured for OpenAI, Gemini, and Anthropic by removing previous provider-side rejections and ensuring OpenAI streaming uses chat.completions.create(stream=True) with a JSON schema response_format when needed.
Changes:
- OpenAI: allow streaming with
response_format, converting structured output types to JSON schema for streaming requests so.parse()is not used. - Gemini and Anthropic: remove
stream + response_formatrejection paths and ensure streaming calls receive the correct provider-specific structured output configuration. - Add and update unit tests to validate the new streaming behavior across the three providers.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/any_llm/providers/openai/base.py |
Updates response_format conversion and streaming behavior to permit structured output while streaming. |
src/any_llm/providers/gemini/base.py |
Removes the unsupported-parameter guard that blocked streaming with response_format. |
src/any_llm/providers/anthropic/utils.py |
Allows output_config generation even when streaming and hardens tool parameter conversion. |
tests/unit/providers/test_openai_base_provider.py |
Adds coverage asserting streaming uses create() (not parse()) for dict and BaseModel response formats. |
tests/unit/providers/test_gemini_provider.py |
Adds coverage asserting streaming passes generation config for JSON schema response formats. |
tests/unit/providers/test_anthropic_provider.py |
Adds coverage asserting streaming passes output_config through to the Anthropic streaming call. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Plain dataclasses are converted to JSON schema dicts since the | ||
| OpenAI SDK's ``.parse()`` only supports Pydantic BaseModel types. | ||
| """ |
There was a problem hiding this comment.
Happy to change this if considered important. let me know
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/any_llm/providers/openai/base.py (1)
85-94: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDo not rewrite
CompletionParams.response_formatin place.This now mutates a caller owned
CompletionParamswhenstream=True. If the same params object is reused for a later non streaming call, the originalBaseModeltype has already been replaced with a dict, so_acompletion()will skip.parse()and silently change behaviour.Proposed fix
- if is_structured_output_type(params.response_format) and ( - params.stream or not issubclass(params.response_format, BaseModel) - ): - params.response_format = { + response_format = params.response_format + if is_structured_output_type(response_format) and ( + params.stream or not issubclass(response_format, BaseModel) + ): + response_format = { "type": "json_schema", "json_schema": { - "name": params.response_format.__name__, - "schema": get_json_schema(params.response_format), + "name": response_format.__name__, + "schema": get_json_schema(response_format), }, } converted_params = params.model_dump(exclude_none=True, exclude={"model_id", "messages"}) + if response_format is not None: + converted_params["response_format"] = response_format🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/any_llm/providers/openai/base.py` around lines 85 - 94, The structured output handling in the OpenAI base provider is mutating caller-owned CompletionParams.response_format in place, which can change later calls when the same params object is reused. Update the logic in the response_format handling path to avoid assigning back onto params.response_format; instead build a local normalized schema payload and pass that through the streaming request path. Keep the original BaseModel type intact so _acompletion() can still detect it and apply .parse() for non-streaming calls.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/any_llm/providers/openai/base.py`:
- Around line 85-94: The structured output handling in the OpenAI base provider
is mutating caller-owned CompletionParams.response_format in place, which can
change later calls when the same params object is reused. Update the logic in
the response_format handling path to avoid assigning back onto
params.response_format; instead build a local normalized schema payload and pass
that through the streaming request path. Keep the original BaseModel type intact
so _acompletion() can still detect it and apply .parse() for non-streaming
calls.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 13fff04e-4461-4281-98f5-d0f8ef24bedb
📒 Files selected for processing (6)
src/any_llm/providers/anthropic/utils.pysrc/any_llm/providers/gemini/base.pysrc/any_llm/providers/openai/base.pytests/unit/providers/test_anthropic_provider.pytests/unit/providers/test_gemini_provider.pytests/unit/providers/test_openai_base_provider.py
💤 Files with no reviewable changes (1)
- src/any_llm/providers/gemini/base.py
Description
Enables streaming chat completions with structured output configuration for providers that support it.
This removes stale
stream + response_formatrejections for Anthropic and Gemini, and updates OpenAI chat completions so structured output can be requested while streaming since it is now supported by all three providers. For OpenAI Pydantic/dataclass response formats, streaming requests convert the type to a JSON schemaresponse_formatbefore callingchat.completions.create(stream=True), preserving chunk streaming without introducing final-response parsing, similar to the other providers.This PR does not change the streaming return contract: streaming still returns chunks only, with no partial or final parsing inside any-llm (since there is no explicit delta accumulator within the library code). This means that parsing, both for pydantic and json schema response formats, will have do be carried out externally.
Note that other providers and/or endpoints likely suffer from the same limitation, but i haven't checked those.
PR Type
Relevant issues
N/A
Checklist
AI Usage Information
When answering questions by the reviewer, please respond yourself, do not copy/paste the reviewer comments into an AI system and paste back its answer. We want to discuss with you, not your AI :)
Summary by CodeRabbit
New Features
Bug Fixes