Skip to content

fix(gemini): grammar-enforce batch structured output regardless of strict flag#2719

Merged
benfrank241 merged 1 commit into
mainfrom
fix/gemini-batch-grammar-enforcement-2699
Jul 15, 2026
Merged

fix(gemini): grammar-enforce batch structured output regardless of strict flag#2719
benfrank241 merged 1 commit into
mainfrom
fix/gemini-batch-grammar-enforcement-2699

Conversation

@benfrank241

Copy link
Copy Markdown
Member

Problem

With LLM_PROVIDER=gemini and HINDSIGHT_API_RETAIN_BATCH_ENABLED=true, the Gemini batch request builder (_openai_body_to_gemini_request in gemini_llm.py) only set the native grammar schema (responseJsonSchema) when json_schema.strict was truthy. But HINDSIGHT_API_LLM_STRICT_SCHEMA defaults to False.

This created a batch-vs-interactive asymmetry: the interactive Gemini path always grammar-enforces structured output via its native response_schema regardless of strict (as its docstring states). strict is an OpenAI concept and is meaningless to Gemini.

As a result, at default config batch requests received only responseMimeType: application/json plus a textual schema hint in the system instruction — no native grammar enforcement. Gemini then intermittently emitted malformed JSON, and since a chunk's facts are parsed as a unit, all facts in the affected chunk were lost.

Fix

Set generationConfig["responseJsonSchema"] = schema whenever a schema is present, regardless of strict, so the batch path mirrors the interactive path. The always-set responseMimeType and the textual schema hint are unchanged. A comment references #2699.

Test

Updated the pure translation unit test in tests/test_gemini_batch.py (test_body_translation_grammar_enforces_schema_without_strict) to build an OpenAI-style body with strict=False and assert the resulting Gemini generationConfig contains responseJsonSchema equal to the schema. This replaces the prior test that asserted the buggy omit-when-not-strict behavior. No DB/LLM — runs fast.

27 passed in 7.26s

Fixes #2699

…rict flag

The Gemini batch request builder only set the native response_schema
(responseJsonSchema) when json_schema.strict was truthy, but
HINDSIGHT_API_LLM_STRICT_SCHEMA defaults to False. The interactive Gemini
path always grammar-enforces via response_schema regardless of strict
(strict is an OpenAI concept, meaningless to Gemini). At default config
batch requests therefore got only responseMimeType + a textual schema hint
and intermittently emitted malformed JSON, dropping every fact in the chunk.

Set responseJsonSchema whenever a schema is present so batch mirrors
interactive. Update the batch translation unit test accordingly.

Fixes #2699
@benfrank241
benfrank241 merged commit d0bbfa1 into main Jul 15, 2026
97 of 101 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemini batch path skips schema grammar enforcement unless strict is set — intermittent malformed JSON, facts silently lost

1 participant