fix(transfer): preserve JSONB and timestamp provenance

[Korayem]

Summary

Whole-bank export/import now preserves the archive's JSONB representation and carried source timestamps instead of guessing or re-deriving them during restore.

The native admin exporter uses decoded JSON/JSONB values, while raw programmatic connections emit serialized JSON text. The manifest now records that provenance explicitly. Import quotes every string in decoded mode and preserves JSON text in serialized mode, so a decoded scalar that happens to contain valid JSON can no longer be mistaken for an object. Legacy v1 bank archives without the optional field default to decoded, matching the released hindsight-admin export-bank producer.

Transfer replay also reapplies documents.created_at and observation event_date inside the existing import transactions. Normal retain/upsert timestamp semantics are unchanged.

Production archive evidence

A restore drill against a real production archive exposed all three regressions:

• llm_requests: all 70/70 ids restored, but 5 JSON-looking scalar output strings became JSON objects under the heuristic importer. The other 23 string outputs were correct and 42 were null.
• Documents: all 23/23 ids, hashes, tags, and retain_params matched, but 2/23 effective dates changed because the importer used restore time for documents.created_at.
• Observations: 18/211 archived event_date values that intentionally differed from both source timestamps were re-derived on restore; none of those 18 source values survived.

Validation

• Added decoded and serialized JSONB provenance regressions, including a decoded scalar containing valid JSON object text.
• Added a focused admin-export boundary regression proving codec-decoded rows are marked decoded in the archive manifest.
• Added legacy-v1 manifest behavior coverage.
• Added exact document created_at round-trip coverage.
• Added observation event_date round-trip coverage.
• uv run pytest tests/test_document_transfer.py tests/test_admin_bank_transfer.py -qq — 23 passed.
• ./scripts/hooks/lint.sh — all lints passed.
• uv run ty check hindsight_api/ — all checks passed.

[benfrank241]

verified this locally and it's a strong, well-evidenced fix — the three restore regressions are real data corruption (decoded JSON-looking scalar strings coerced to objects, documents.created_at reset to restore time, observation event_date re-derived), and the production-archive numbers (5/28 outputs, 2/23 dates, 18/211 event_dates) make it concrete. the fix is careful: optional bank_rows_json_encoding manifest field (no schema-version bump), legacy v1 archives default to decoded to match the released admin producer, and timestamps reapplied inside the existing import txns. 22/22 transfer tests pass on the branch, rebased on #2685.

flagging for @nicoloboschi to own the merge rather than self-merging — not a fix-quality concern, purely subsystem coordination:

1. it commits to a new archive-format field + provenance semantics (decoded vs serialized, legacy default) — a forward-compat decision that's locked once archives start carrying it. worth your sign-off on the format shape.
2. it touches transfer/{export,importer,schema}.py + admin/cli.py — the same surface as fix(engine): consolidate duplicated engine definitions #2691 (transfer dedup refactor, currently in your queue) and the merged fix(engine): clarify causal link compatibility #2685. merging this now would force a rebase on fix(engine): consolidate duplicated engine definitions #2691, so you'll want to sequence the two.

no objection to the change itself — just want the format commitment + the #2691 ordering to be your call.

[nicoloboschi]

@Korayem please rebase

[Korayem]

@Korayem please rebase

on it

[Korayem]

@Korayem please rebase

Rebased onto current main (0c38d46ee). The overlap with the transfer dedup changes was resolved by keeping upstream's shared HISTORY_TABLES constant while preserving archive JSON provenance through history-row restore. Focused transfer tests pass (23/23), lint and type checks are clean, and the new head is e03f8d357.

[Korayem]

@nicoloboschi check now