Skip to content

feat(retain): optional fail-on-extraction-errors flag (issue #2700)#2721

Merged
benfrank241 merged 3 commits into
mainfrom
fix/fail-on-extraction-errors-2700
Jul 15, 2026
Merged

feat(retain): optional fail-on-extraction-errors flag (issue #2700)#2721
benfrank241 merged 3 commits into
mainfrom
fix/fail-on-extraction-errors-2700

Conversation

@benfrank241

Copy link
Copy Markdown
Member

Problem: silent success on dropped facts

When batch fact extraction fails for a chunk during retain, the error is recorded
non-fatally (into a RetainExtractionErrors accumulator, persisted as
extraction_errors_count / extraction_errors_sample in the operation's
result_metadata) but the retain async operation is still marked
status='completed' unconditionally. Nothing reads those counters to change the
terminal status, so a retain that dropped all of its facts still reports a
clean success — the worst failure mode for a memory system, because the caller
has no signal that memories were lost.

The fix: an opt-in escape hatch (default off)

Adds a new static, server-level config flag
HINDSIGHT_API_FAIL_ON_EXTRACTION_ERRORS (default False, so current
behavior is 100% preserved). When it is True and the retain operation
accumulated any extraction errors (extraction_errors_count > 0), the operation
is marked failed with an informative error_message (includes the error count
and mentions extraction errors) instead of completed.

The decision is made at the single retain completion point
(_mark_operation_completed), reading the extraction_errors_count that
_write_retain_outcome_metadata already persisted before completion. When the
flag is off, the code path is unchanged.

Changes

  • config.py: ENV_FAIL_ON_EXTRACTION_ERRORS, DEFAULT_FAIL_ON_EXTRACTION_ERRORS = False,
    a static fail_on_extraction_errors: bool field wired in from_env().
  • memory_engine.py: _mark_operation_completed now marks failed (with an
    informative message) when the flag is on and the persisted extraction-error
    count is > 0; otherwise unchanged.
  • .env.example (+ synced embed bundle copy) and configuration.md doc row.
  • Tests: config parsing (default False / parses True) and the completion-status
    behavior (fails when flag on + errors, stays completed when flag off, stays
    completed when flag on but zero errors).

Deferred follow-ups (intentionally out of scope)

To keep this change self-contained, the following are left for sibling PRs and
do not touch API response models, generated clients, webhooks, or the DB status
enum:

  • Surface extraction_errors_count in the operation-status API response.
  • Add a distinct completed_with_errors terminal status (needs a DB enum value).
  • Add an extraction-error field on RetainEventData for webhook consumers.
  • Add a Prometheus counter for extraction errors.

Note: the root cause of the parse failures themselves (why extraction fails on
certain chunks) is handled by sibling PRs for #2699 and #2701; this PR is purely
about not silently reporting success when facts were dropped.

Fixes #2700

@benfrank241
benfrank241 merged commit ec3b415 into main Jul 15, 2026
96 of 100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retain extraction failures are completely silent — operation reports completed even when chunks lose all facts

1 participant