fix(gemini): grammar-enforce batch structured output regardless of strict flag#2719
Merged
Merged
Conversation
…rict flag The Gemini batch request builder only set the native response_schema (responseJsonSchema) when json_schema.strict was truthy, but HINDSIGHT_API_LLM_STRICT_SCHEMA defaults to False. The interactive Gemini path always grammar-enforces via response_schema regardless of strict (strict is an OpenAI concept, meaningless to Gemini). At default config batch requests therefore got only responseMimeType + a textual schema hint and intermittently emitted malformed JSON, dropping every fact in the chunk. Set responseJsonSchema whenever a schema is present so batch mirrors interactive. Update the batch translation unit test accordingly. Fixes #2699
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
LLM_PROVIDER=geminiandHINDSIGHT_API_RETAIN_BATCH_ENABLED=true, the Gemini batch request builder (_openai_body_to_gemini_requestingemini_llm.py) only set the native grammar schema (responseJsonSchema) whenjson_schema.strictwas truthy. ButHINDSIGHT_API_LLM_STRICT_SCHEMAdefaults toFalse.This created a batch-vs-interactive asymmetry: the interactive Gemini path always grammar-enforces structured output via its native
response_schemaregardless ofstrict(as its docstring states).strictis an OpenAI concept and is meaningless to Gemini.As a result, at default config batch requests received only
responseMimeType: application/jsonplus a textual schema hint in the system instruction — no native grammar enforcement. Gemini then intermittently emitted malformed JSON, and since a chunk's facts are parsed as a unit, all facts in the affected chunk were lost.Fix
Set
generationConfig["responseJsonSchema"] = schemawhenever a schema is present, regardless ofstrict, so the batch path mirrors the interactive path. The always-setresponseMimeTypeand the textual schema hint are unchanged. A comment references #2699.Test
Updated the pure translation unit test in
tests/test_gemini_batch.py(test_body_translation_grammar_enforces_schema_without_strict) to build an OpenAI-style body withstrict=Falseand assert the resulting GeminigenerationConfigcontainsresponseJsonSchemaequal to the schema. This replaces the prior test that asserted the buggy omit-when-not-strict behavior. No DB/LLM — runs fast.Fixes #2699