fix(retain): preserve structured envelope fields across oversized chunks

[r266-tech]

Summary

• split oversized structured records inside one unambiguous payload path and re-wrap every fragment as valid JSON
• preserve message, tool-call, and path context such as role, timestamp, id, and path, while retaining the plain-text fallback for ambiguous shapes
• bound recursion, fragment count, output expansion, and serialized wrapper overhead so chunks remain lossless and idempotent

Testing

• uv run pytest tests/test_chunking.py -q (44 passed)
• uv run pytest tests/test_structured_chunk_size_retain.py -q -n 0
• uv run --frozen ty check hindsight_api
• ./scripts/hooks/lint.sh

Fixes #2548

[benfrank241]

Thanks @r266-tech — thorough work, and the test coverage (losslessness, idempotency, plain-text fallback for ambiguous shapes, envelope preservation) is exactly right for this kind of change.

Verified locally on the PR head:

• tests/test_chunking.py: 44/44 pass
• tests/test_structured_chunk_size_retain.py: 1/1 pass (test_jsonl_line_over_chunk_size_retains_without_collision)

Routing the merge decision to @nicoloboschi rather than merging directly, for two reasons:

1. This is a feature, not a bug fix. Smarter json chunking to prevent loss of critical context #2548 is filed as a feature request ("Smarter json chunking"), and the PR implements new structured-splitting behavior (~500 new lines: envelope-key heuristics, recursive JSON-aware splitting with re-wrapping, and the depth/fragment/output-expansion bounds).
2. It changes the retain chunking path, which is accuracy-sensitive (feeds extraction/attribution quality on the retain leaderboard + longmemeval/locomo). The envelope-key sets and shape-detection (_MESSAGE_*_ENVELOPE_KEYS, _TOOL_CALL_TYPES, etc.) are effectively contract decisions worth an owner’s eye, especially how they behave on session formats other than the Claude/OpenAI shapes in the tests.

No blocking concerns from me — the diff is well-bounded (recursion/fragment/expansion caps), lint + ty are clean per the PR description and my run, and it degrades to the existing plain-text split for anything ambiguous. Flagging it as a design-review item so the heuristics and forward-compat are your call. cc @nicoloboschi