[Bug]: Azure Blog Storage Connector: "Ingested" status badge no longer shown in "Browse Files" dialog for documents already ingested

[mpawlow]

OpenRAG Version

0.6.2

Deployment Method

Local development (make dev)

Operating System

Ubuntu 24.04.4 LTS

Python Version

Python 3.13.14

Affected Area

Connectors (Google Drive, OneDrive, SharePoint)

Bug Description

• "Ingested" status badge no longer shown in "Browse Files" dialog for documents already ingested

Steps to Reproduce

• Go to Settings page
• Setup Azure Blob Storage connector.
• Click Add Knowledge button to go to the Add from Azure Storage pane
• Ingest all files from a container
• Go to Knowledge page to ensure all files are successfully ingested
• Click Add Knowledge button to go to the Add from Azure Storage pane
• Click Browse Files button
  • BUG: "Ingested" badge is not shown for files already ingested

Expected Behavior

• "Ingested" status badge to be shown in "Browse Files" dialog for documents already ingested

Actual Behavior

• "Ingested" status badge no longer shown in "Browse Files" dialog for documents already ingested
• This is a regression

Relevant Logs

N/A

Screenshots

Expected

Additional Context

• For Azure Blog Storage connector setup instructions, see offline PDF file (OpenRAG - Azure Blob Storage Connector - Manual Testing Instructions.pdf)

Checklist

• I have searched existing issues to ensure this bug hasn't been reported before.
• I have provided all the requested information.

[mpawlow]

• ⚠️ Possibly related issue: [Bug]: Azure Blog Storage Connector: Optimized sync facility no longer prevents documents that were previously ingested from being re-ingested #2085

[mpawlow]

Background Info

• How the badge actually breaks: browse_connection_files → get_synced_file_ids_for_connector (src/api/connectors.py) queries OpenSearch via the user's scoped (DLS) client to find already-ingested file IDs.
• Critically, it fails open to "nothing is ingested" - any exception is caught, logged, and swallowed, returning empty sets.
• That exactly matches the symptom: files list fine, badges just vanish.
• ℹ️ Unverified: Regression may have been caused by 1591ee4

[hunterxtang]

I am unable to reproduce this bug on my end

[mpawlow]

Status Update

• ✅ Problem not reproducible on my first Fyre VM
• ❌ Problem still reproducible on my second Fyre VM

Backend Logs

[2026-07-14T20:25:56.152583Z] [ERROR  ] [connectors.py:170             ] Failed to get synced file IDs
                                                                         - connector_type: azure_blob
                                                                         - error: RequestError(400, 'search_phase_execution_exception', 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [connector_file_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.')
                                                                         - method: GET
                                                                         - request_id: d5bcc685-382b-4857-87de-0c965b3e38fb
                                                                         - path: /connectors/azure_blob/0be24cfb-4277-42cd-9831-f689734fe006/browse
[2026-07-14T20:25:56.162990Z] [ERROR  ] [connectors.py:285             ] Failed to build id→modified_time map
                                                                         - connector_type: azure_blob
                                                                         - error: RequestError(400, 'search_phase_execution_exception', 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [connector_file_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.')
                                                                         - method: GET
                                                                         - request_id: d5bcc685-382b-4857-87de-0c965b3e38fb
                                                                         - path: /connectors/azure_blob/0be24cfb-4277-42cd-9831-f689734fe006/browse

[mpawlow]

Root Cause Analysis

• The second Fyre VM documents index in OpenSearch has connector_file_id mapped as text (dynamically-mapped, with a .keyword subfield), but src/config/settings.py:670 declares it as keyword in the code's INDEX_BODY.
• That mapping is only applied when the index is created
• Since the index predates (or was recreated without) that explicit mapping, OpenSearch fell back to dynamic mapping the first time a doc wrote that field, resulting in a text field instead.
• get_synced_file_ids_for_connector() in src/api/connectors.py:107 runs a terms aggregation directly on connector_file_id, which OpenSearch refuses on a text field
• The exception is caught at connectors.py:169-175 and silently returns empty lists, so the Azure Blob browse endpoint (connectors.py:2271) treats every file as not-ingested — hence no badges, even though ingestion succeeded.
• ℹ️ This isn't Azure-Blob-specific code — it's the first connector on this environment to actually populate connector_file_id and hit that aggregation.
• ❓ Unknown how the OpenSearch index got out of sync with the latest schema

Work-around

• Data-preserving: created documents_v2 with the corrected mapping, _reindexed all 406 docs (0 failures), deleted the old documents index, and aliased documents → documents_v2.
• No backend restart or code change was needed.
• ✅ Badges now show correctly in the UI and sync behavior works as expected.

[mpawlow]

Root Cause Analysis - Part 2

• There is a self-healing mechanism for exactly this class of index field drift, but it has a gap that let connector_file_id slip through.
• The mechanism (src/utils/opensearch_init.py:206-227, _ensure_opensearch_index): on every backend startup, if the documents index already exists, the code does not recreate it with the current INDEX_BODY — it only calls _ensure_keyword_mappings() (line 216-220) to patch in a hardcoded list of fields as keyword if they're missing from the mapping:

  await _ensure_keyword_mappings(
      clients.opensearch, index_name,
      ["allowed_users", "allowed_groups", "allowed_principals", "ingest_run_id"],
  )

• The gap: connector_file_id was added to INDEX_BODY back on 2026-05-29 (commit 567231c), but was never added to this self-heal list.
• So on any index that already existed (or got created/reset while running older code, before this field existed), the field was left unmapped.
• The first time a chunk was indexed with connector_file_id set, OpenSearch's default dynamic string mapping kicked in and typed it text + a .keyword subfield — silently, with no warning, because _ensure_keyword_mappings only checks fields you explicitly pass it.
• ⚠️ Note: Even for fields that are in the list, the function only logs a warning if they're already wrong-typed (line 46-52) — it can't fix a field that's already drifted, since OpenSearch doesn't allow changing a field's type in place.
• That's exactly why we needed the reindex+alias-swap approach rather than a simple put_mapping.
• 👉 This index was almost certainly created (or last reset) by a backend build that predated connector_file_id in INDEX_BODY, and no restart since has repaired it because the self-heal list never covered that field.

[mpawlow]

Status Update

• ✅ Created a PR to address old OpenSearch indices that require the fallback (edge case)
  • feat: Azure Blog Connector - "Ingested" status badge is missing #2099