fix: japanese file names (backport of #2053)

[edwinjosechittilappilly]

Summary

Backports #2053 from release-saas-ga-0.6.2 to main.

Fixes handling of non-ASCII (e.g. Japanese) filenames for bucket connectors by threading a connector_file_id through document indexing/ingest-token context, independent of the display filename.

Note on conflict resolution: cherry-pick conflicted in src/services/langflow_file_service.py and src/services/langflow_ingest_token_service.py against an unrelated parser/chunk_size/chunk_overlap feature already on main. Both are additive fields on DocumentIndexContext; resolved by keeping both sets of fields together.

Test plan

• pytest tests/unit/test_document_index_writer_connector_file_id.py tests/unit/test_resolve_document_id_connector_file_id.py tests/unit/connectors/test_bucket_connector_unicode_file_ids.py tests/unit/test_langflow_ingest_callback.py tests/unit/test_connector_processor_filename_dedupe.py tests/unit/connectors/test_update_connector_metadata_filename.py — 47 passed
• CI passes

Summary by CodeRabbit

• Bug Fixes

  • Improved connector-file ingestion, chunk cleanup, and metadata updates by consistently using a connector-specific identifier (including connector_file_id.keyword for older mappings).
  • Enhanced handling of renamed files so updates reliably apply to all matching chunks.
  • Ensured bucket-style connector IDs and non-ASCII object keys round-trip correctly.
  • Extended ingestion tokens to carry and restore the connector identifier; stable ASCII document IDs are generated when needed.

• Tests

  • Added/updated regression coverage for connector identifier propagation, Unicode bucket/blob keys, metadata update matching, and token round-tripping.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bf69d998-87d3-4a80-94bc-8059dc41d9bd

📥 Commits

Reviewing files that changed from the base of the PR and between 6d2db71 and 01d6ab5.

📒 Files selected for processing (2)

• src/connectors/service.py
• src/models/processors.py

🚧 Files skipped from review as they are similar to previous changes (2)

• src/models/processors.py
• src/connectors/service.py

---

Walkthrough

Connector ingestion now preserves raw connector_file_id values separately from hashed document_id values, propagates them through Langflow context and tokens, and expands OpenSearch matching for ingestion, verification, deletion, and metadata updates.

Changes

Connector identity propagation

Layer / File(s)	Summary
Stable connector identity plumbing src/services/langflow_file_service.py, src/services/document_index_writer.py, src/services/langflow_ingest_token_service.py, tests/unit/...	Langflow hashes connector identifiers into ASCII-safe document IDs while carrying raw connector IDs through ingestion context, JWT tokens, and indexed documents. Tests cover hashing, fallback behavior, token round trips, document propagation, and Unicode bucket keys.
OpenSearch matching and connector processing src/connectors/service.py, src/models/processors.py, src/utils/acl_utils.py, tests/unit/...	Connector ingestion and metadata operations match document_id, connector_file_id, and connector_file_id.keyword; existence checks accept a selectable field, and cleanup paths use the expanded selectors.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ConnectorService
  participant LangflowFileService
  participant DocumentIndexWriter
  participant OpenSearch
  ConnectorService->>LangflowFileService: Upload with connector_file_id
  LangflowFileService->>DocumentIndexWriter: Pass raw connector ID and hashed document ID
  DocumentIndexWriter->>OpenSearch: Index chunk identifiers
  ConnectorService->>OpenSearch: Update metadata using identifier match clauses
  ConnectorService->>OpenSearch: Verify and delete connector chunks

Loading

Possibly related PRs

• langflow-ai/openrag#2053: Threads connector_file_id through Langflow ingestion and connector metadata handling.
• langflow-ai/openrag#2096: Refactors bucket ingestion around connector identifiers, keyword matching, and hashed document IDs.
• langflow-ai/openrag#2017: Changes connector existence queries and post-ingestion verification.

Suggested reviewers: mpawlow, ricofurtado

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 23.81% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main fix for Japanese/non-ASCII file names and notes it is a backport.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch backport/2053-japanese-file-names

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}