fix(retain): recover recoverable batch-extraction JSON instead of dropping facts

[benfrank241]

Problem

The Batch API result parser in hindsight-api-slim/hindsight_api/engine/retain/fact_extraction.py (in extract_facts_from_contents_batch_api) parsed each chunk's response with a bare json.loads(content_str). On json.JSONDecodeError it logged the failure, recorded the chunk with fact_count=0, and continued — so all facts in that chunk were permanently lost, with no recovery.

Meanwhile a lenient parser already existed and was not used on this path: parse_llm_json() in hindsight_api/engine/llm_wrapper.py. It strips markdown code fences (json … ) and, on failure, retries after replacing embedded control characters (\x00-\x1f, \x7f) — the exact quirks some models (e.g. Gemini) emit inside JSON string values.

Production evidence: transient Gemini batch parse failures that succeeded on manual reprocessing.

Fix (layer 1 — the cheap, immediate one)

Swap the bare json.loads(content_str) in the batch result path for the existing parse_llm_json(content_str) helper:

try:
    # #2701: use the lenient parser (strips markdown fences, scrubs
    # embedded control chars) so recoverable batch responses — e.g.
    # transient Gemini quirks — aren't dropped along with all their facts.
    extraction_response_json = parse_llm_json(content_str)
except json.JSONDecodeError as e:
    message = f"{custom_id}: failed to parse JSON: {e}"
    ...

parse_llm_json returns the parsed object and re-raises json.JSONDecodeError if the text is still unrecoverable, so the existing error path is preserved unchanged (record custom_id: failed to parse JSON, append ChunkMetadata(fact_count=0, …), continue). Only the successful-recovery path is new.

Tests

Added to tests/test_batch_api.py (pure mocked, no LLM):

• test_batch_api_recovers_fenced_and_control_char_json — batch content wrapped in ```json fences and containing a raw \x01 control character inside a string value (asserted unparseable by bare `json.loads`); now yields `fact_count == 1` and the facts are present rather than lost.
• test_batch_api_unparseable_json_still_records_error — genuinely unparseable content still records the error and yields fact_count == 0 (behavior preserved).

Both pass:

tests/test_batch_api.py::test_batch_api_recovers_fenced_and_control_char_json PASSED
tests/test_batch_api.py::test_batch_api_unparseable_json_still_records_error PASSED

Follow-up (out of scope here)

This is only "layer 1" (in-place lenient re-parse). A "layer 2" — resubmitting a still-unparseable chunk through the interactive (non-batch) extraction path for a full retry — is a larger change and is left as a documented follow-up.

Fixes #2701