fix: Skip deleted connector files; return orphan IDs#1592
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR adds orphan detection and deletion for connector-indexed chunks, integrates reconciliation into connector sync flows and preview endpoints, makes processors skip deleted remote files, updates connector filename metadata propagation, modernizes typing/logging, adds frontend preview-confirm UI, tests, and adjusts .gitignore. ChangesOrphan File Cleanup and Connector Resilience
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested labels
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
backend
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR improves connector sync robustness by skipping missing/deleted source files instead of aborting syncs, and by adding an orphan-reconciliation helper that deletes chunks for documents no longer present remotely.
Changes:
- Add
TaskStatus.SKIPPEDand handle deleted-at-source files during processing without failing the overall sync. - Add
delete_chunks_by_document_idsandreconcile_orphans_for_connector_typeto delete orphaned OpenSearch chunks and return orphan IDs. - Update connector metadata update script to rewrite the indexed
filenamefield.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
src/models/tasks.py |
Adds SKIPPED task status to represent skipped file processing outcomes. |
src/models/processors.py |
Catches missing-source-file errors from connectors and marks file tasks as skipped instead of raising. |
src/api/documents.py |
Adds a bulk delete helper for OpenSearch chunks by document_id. |
src/api/connectors.py |
Adds orphan reconciliation that lists remote IDs, deletes orphans, and returns orphan IDs. |
src/connectors/service.py |
Updates OpenSearch metadata update script to persist filename updates; minor formatting cleanup. |
tests/unit/api/test_delete_chunks_by_document_ids.py |
Adds unit tests covering the new bulk delete helper behavior. |
tests/unit/api/test_reconcile_orphans_for_connector_type.py |
Adds unit tests for orphan reconciliation logic (currently misaligned with new return type). |
tests/unit/connectors/test_update_connector_metadata_filename.py |
Adds unit tests to pin filename rewrite behavior in metadata updates. |
.gitignore |
Adjusts ignored AI-tool related files (not described in PR metadata). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| existing_file_ids=[], | ||
| ) | ||
|
|
||
| assert deleted == 0 |
| existing_file_ids=["a", "b"], | ||
| ) | ||
|
|
||
| assert deleted == 0 |
| existing_file_ids=["a", "b"], | ||
| ) | ||
|
|
||
| assert deleted == 0 |
| existing_file_ids=["a", "b"], | ||
| ) | ||
|
|
||
| assert deleted == 0 |
| existing_file_ids=["a", "b", "c"], | ||
| ) | ||
|
|
||
| assert deleted == 7 |
| existing_file_ids=["a", "b", "c"], | ||
| ) | ||
|
|
||
| assert deleted == 0 |
| existing_file_ids=["a", "b"], | ||
| ) | ||
|
|
||
| assert deleted == 0 |
Comment on lines
+371
to
+382
| # Reconcile orphans (files deleted at the source) before re-syncing. | ||
| # Strict gating: skip when sync is capped — we'd see a partial remote | ||
| # listing and delete legitimate files. | ||
| if body.max_files is None: | ||
| await reconcile_orphans_for_connector_type( | ||
| connector_type=connector_type, | ||
| user_id=user.user_id, | ||
| connector_service=connector_service, | ||
| session_manager=session_manager, | ||
| jwt_token=jwt_token, | ||
| existing_file_ids=existing_file_ids, | ||
| ) |
Comment on lines
+524
to
+538
| except (FileNotFoundError, ValueError) as e: | ||
| msg = str(e).lower() | ||
| if "not found" in msg or "404" in msg: | ||
| logger.warning( | ||
| "File no longer exists at source — skipping", | ||
| file_id=file_id, | ||
| connection_id=self.connection_id, | ||
| error=str(e), | ||
| ) | ||
| file_task.status = TaskStatus.SKIPPED | ||
| file_task.result = {"status": "skipped", "reason": "deleted_at_source"} | ||
| file_task.updated_at = time.time() | ||
| upload_task.successful_files += 1 | ||
| return | ||
| raise |
Comment on lines
+522
to
+537
| try: | ||
| document = await connector.get_file_content(file_id) | ||
| except (FileNotFoundError, ValueError) as e: | ||
| msg = str(e).lower() | ||
| if "not found" in msg or "404" in msg: | ||
| logger.warning( | ||
| "File no longer exists at source — skipping", | ||
| file_id=file_id, | ||
| connection_id=self.connection_id, | ||
| error=str(e), | ||
| ) | ||
| file_task.status = TaskStatus.SKIPPED | ||
| file_task.result = {"status": "skipped", "reason": "deleted_at_source"} | ||
| file_task.updated_at = time.time() | ||
| upload_task.successful_files += 1 | ||
| return |
github-actions
Bot
added
bug
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/api/connectors.py (1)
1-1:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUse modern type hint syntax instead of deprecated
typingimports.Python 3.9+ supports
list,dict, andX | Nonesyntax directly. The deprecatedList,Dict, andOptionalimports fromtypingtrigger static analysis warnings (UP006, UP045, UP035).♻️ Proposed fix
-from typing import Any, Dict, List, Optional +from typing import AnyThen update the function signature at lines 80-87:
async def reconcile_orphans_for_connector_type( connector_type: str, user_id: str, connector_service, session_manager, - jwt_token: Optional[str], - existing_file_ids: List[str], -) -> List[str]: + jwt_token: str | None, + existing_file_ids: list[str], +) -> list[str]:Based on learnings, static analysis tools flagged UP006, UP045, and UP035 violations for deprecated typing imports.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/api/connectors.py` at line 1, Replace deprecated typing imports by removing List, Dict, and Optional from the import line and use built-in generics and union syntax instead (e.g., list[], dict[], and X | None) in the code; specifically update the import statement currently reading "from typing import Any, Dict, List, Optional" to only import Any if needed and change all function signatures and annotations that use List, Dict, and Optional (including the function signature referenced around lines 80-87) to use built-in list[...] and dict[...] types and the new union/nullable syntax (Type | None) so static-analysis warnings UP006/UP045/UP035 are resolved.
🧹 Nitpick comments (1)
src/models/processors.py (1)
524-526: ⚡ Quick winFragile exception handling: broad ValueError with string matching.
Catching
ValueErrorand checking for "not found" or "404" in the message is fragile:
ValueErroris a very broad exception type used for many unrelated errors- String matching on error messages can break if connectors change their error text
- This pattern could mask other
ValueErrorissues unrelated to missing filesConsider defining connector-specific exception types (e.g.,
ConnectorFileNotFoundError) or using more specific exception matching.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/models/processors.py` around lines 524 - 526, The current except block catches ValueError and relies on string-matching (variable e / msg) to detect missing files which is fragile; define a connector-specific exception (e.g., ConnectorFileNotFoundError) in the connectors module, update all connector code that currently raises ValueError for missing resources to raise ConnectorFileNotFoundError (or wrap the original exception), and then replace the except (FileNotFoundError, ValueError) as e branch with except (FileNotFoundError, ConnectorFileNotFoundError) as e, removing the "not found"/"404" string checks so the code reliably handles missing-file cases without matching error text.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/api/connectors.py`:
- Around line 80-167: The function reconcile_orphans_for_connector_type
currently declares -> List[str] and returns orphans (List[str]) but should
return the integer count returned by delete_chunks_by_document_ids; update the
signature to -> int, change the final return to return the deleted variable
instead of orphans, and ensure exceptions still return 0 to match the int
contract; also replace typing imports/usages (List, Optional, Dict) with modern
builtins (e.g., list, Optional[str] or str | None) throughout this function to
avoid deprecated types — look for reconcile_orphans_for_connector_type,
delete_chunks_by_document_ids, and session_manager.get_user_opensearch_client to
locate the relevant code.
In `@src/models/processors.py`:
- Line 536: The current code increments upload_task.successful_files for skipped
files which misrepresents outcomes; update the UploadTask model (class
UploadTask in src/models/tasks.py) to add a skipped_files counter, increment
upload_task.skipped_files for skip cases instead of
upload_task.successful_files, and remove the successful_files increment from the
skipped-path; also update any status/reporting/serialization that reads
successful_files to reflect the new skipped_files field so metrics remain
accurate.
- Around line 522-538: The skip branch in the try/except around
connector.get_file_content sets file_task.status to TaskStatus.SKIPPED and
returns early but never increments upload_task.processed_files; update the code
around connector.get_file_content (the try/except handling the
FileNotFoundError/ValueError for connector.get_file_content) to ensure
upload_task.processed_files is always incremented by adding a finally block (or
moving the increment into a shared exit path) so that when you set
file_task.status = TaskStatus.SKIPPED and file_task.result = {...} you still
increment upload_task.processed_files and update timestamps before returning;
make the change adjacent to file_task.updated_at and
upload_task.successful_files to keep behavior consistent with other processors
like DocumentFileProcessor.
In `@tests/unit/api/test_reconcile_orphans_for_connector_type.py`:
- Around line 70-88: The tests expect an int but
reconcile_orphans_for_connector_type returns List[str]; update the tests to
assert against the returned list (or its length) instead of integers: in
test_empty_existing_file_ids_returns_zero_without_calls assign the result to a
variable like orphans (instead of deleted) and assert orphans == [] (and keep
service.connection_manager.list_connections.assert_not_awaited()), and in the
other test replace assert deleted == 7 with either assert len(orphans) == 7 or
assert orphans == [<expected_ids>] as appropriate; ensure all other assertions
in these tests reference the new variable name and list semantics.
---
Outside diff comments:
In `@src/api/connectors.py`:
- Line 1: Replace deprecated typing imports by removing List, Dict, and Optional
from the import line and use built-in generics and union syntax instead (e.g.,
list[], dict[], and X | None) in the code; specifically update the import
statement currently reading "from typing import Any, Dict, List, Optional" to
only import Any if needed and change all function signatures and annotations
that use List, Dict, and Optional (including the function signature referenced
around lines 80-87) to use built-in list[...] and dict[...] types and the new
union/nullable syntax (Type | None) so static-analysis warnings
UP006/UP045/UP035 are resolved.
---
Nitpick comments:
In `@src/models/processors.py`:
- Around line 524-526: The current except block catches ValueError and relies on
string-matching (variable e / msg) to detect missing files which is fragile;
define a connector-specific exception (e.g., ConnectorFileNotFoundError) in the
connectors module, update all connector code that currently raises ValueError
for missing resources to raise ConnectorFileNotFoundError (or wrap the original
exception), and then replace the except (FileNotFoundError, ValueError) as e
branch with except (FileNotFoundError, ConnectorFileNotFoundError) as e,
removing the "not found"/"404" string checks so the code reliably handles
missing-file cases without matching error text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 82bb40b1-43af-470d-bec1-0b2a210c26e0
📒 Files selected for processing (10)
.gitignoresrc/api/connectors.pysrc/api/documents.pysrc/connectors/service.pysrc/models/processors.pysrc/models/tasks.pytests/unit/api/test_delete_chunks_by_document_ids.pytests/unit/api/test_reconcile_orphans_for_connector_type.pytests/unit/connectors/__init__.pytests/unit/connectors/test_update_connector_metadata_filename.py
| file_task.status = TaskStatus.SKIPPED | ||
| file_task.result = {"status": "skipped", "reason": "deleted_at_source"} | ||
| file_task.updated_at = time.time() | ||
| upload_task.successful_files += 1 |
Contributor
There was a problem hiding this comment.
Consider whether skipped files should count as successful.
Incrementing successful_files for a skipped file is semantically confusing—a skipped file isn't truly "successful." This could mislead users or monitoring systems about the actual success rate of the sync operation.
Consider either:
- Adding a separate
skipped_filescounter to theUploadTaskmodel (insrc/models/tasks.py) - Not incrementing any success/failure counter for skipped files
- If the intent is "successfully handled by not aborting," document this explicitly in the code
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/models/processors.py` at line 536, The current code increments
upload_task.successful_files for skipped files which misrepresents outcomes;
update the UploadTask model (class UploadTask in src/models/tasks.py) to add a
skipped_files counter, increment upload_task.skipped_files for skip cases
instead of upload_task.successful_files, and remove the successful_files
increment from the skipped-path; also update any status/reporting/serialization
that reads successful_files to reflect the new skipped_files field so metrics
remain accurate.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/api/documents.py`:
- Around line 140-141: The import block in src/api/documents.py is unsorted;
reorder the imports so they follow alphabetical order to satisfy Ruff (e.g.,
ensure the line importing IBM_AUTH_ENABLED and clients as app_clients and the
line importing init_index are sorted correctly). Update the top-of-file import
group so names are alphabetized and run ruff check --fix or re-run the linter to
confirm the import order is valid.
- Around line 125-136: The delete_chunks_by_document_ids function currently
lacks input validation, error handling, and logging; update the function
(delete_chunks_by_document_ids) to validate that opensearch_client is not None
and index_name is a non-empty string (and keep the existing empty document_ids
short-circuit), wrap the call to opensearch_client.delete_by_query in a
try/except to catch exceptions (network/auth/index errors), log any exception
with the same logger and style used by delete_documents_by_filename_core, and on
error return 0 (or the safest fallback) instead of letting the exception
propagate; also log successful deletion counts before returning.
In `@src/connectors/service.py`:
- Around line 1-7: The import block in this module is unsorted (Ruff I001);
reorder imports into standard isort groups (stdlib first: typing.Any), then
third-party (none here), then local application imports, and ensure relative
imports stay grouped—so place "from typing import Any" first, then utility
imports "from utils.file_utils import get_file_extension,
clean_connector_filename" and "from utils.logging_config import get_logger", and
finally the local connector imports "from .base import BaseConnector,
ConnectorDocument" and "from .connection_manager import ConnectionManager"; run
ruff --fix or isort to apply and verify the change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ae4ea5b4-087d-45f7-83d2-3a45f71ca3aa
📒 Files selected for processing (4)
src/api/connectors.pysrc/api/documents.pysrc/connectors/service.pysrc/models/tasks.py
🚧 Files skipped from review as they are similar to previous changes (2)
- src/models/tasks.py
- src/api/connectors.py
Comment on lines
+125
to
+136
| async def delete_chunks_by_document_ids( | ||
| document_ids: list[str], | ||
| opensearch_client, | ||
| index_name: str, | ||
| ) -> int: | ||
| """Bulk delete OpenSearch chunks by document_id. Returns deleted count.""" | ||
| if not document_ids: | ||
| return 0 | ||
| body = {"query": {"terms": {"document_id": document_ids}}} | ||
| res = await opensearch_client.delete_by_query(index=index_name, body=body, conflicts="proceed") | ||
| return res.get("deleted", 0) | ||
|
|
Contributor
There was a problem hiding this comment.
Add error handling and input validation.
The function lacks:
- Error handling: If
delete_by_queryfails (network issue, auth failure, index not found), the exception propagates uncaught. This could abort the caller's sync operation unexpectedly. - Input validation: No checks for
opensearch_clientbeingNoneorindex_namebeing empty, which could produce obscure errors. - Logging: Unlike
delete_documents_by_filename_core(lines 74-77), this function doesn't log deletions, creating an inconsistent observability pattern.
🛡️ Proposed fix adding error handling, validation, and logging
async def delete_chunks_by_document_ids(
document_ids: list[str],
opensearch_client,
index_name: str,
) -> int:
"""Bulk delete OpenSearch chunks by document_id. Returns deleted count."""
if not document_ids:
return 0
+ if not opensearch_client:
+ logger.warning("delete_chunks_by_document_ids called with no client; returning 0")
+ return 0
+ if not index_name:
+ logger.warning("delete_chunks_by_document_ids called with no index_name; returning 0")
+ return 0
+
+ try:
- body = {"query": {"terms": {"document_id": document_ids}}}
- res = await opensearch_client.delete_by_query(index=index_name, body=body, conflicts="proceed")
- return res.get("deleted", 0)
+ body = {"query": {"terms": {"document_id": document_ids}}}
+ res = await opensearch_client.delete_by_query(index=index_name, body=body, conflicts="proceed")
+ deleted_count = res.get("deleted", 0)
+ logger.info(f"Deleted {deleted_count} orphan chunks by document_id", count=deleted_count, document_ids=len(document_ids))
+ return deleted_count
+ except Exception as e:
+ logger.error("Failed to delete chunks by document_ids", error=str(e), document_ids=len(document_ids))
+ return 0🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/api/documents.py` around lines 125 - 136, The
delete_chunks_by_document_ids function currently lacks input validation, error
handling, and logging; update the function (delete_chunks_by_document_ids) to
validate that opensearch_client is not None and index_name is a non-empty string
(and keep the existing empty document_ids short-circuit), wrap the call to
opensearch_client.delete_by_query in a try/except to catch exceptions
(network/auth/index errors), log any exception with the same logger and style
used by delete_documents_by_filename_core, and on error return 0 (or the safest
fallback) instead of letting the exception propagate; also log successful
deletion counts before returning.
github-actions
Bot
added
bug
Collaborator
Author
Screen.Recording.2026-05-13.at.12.05.25.PM.mov |
Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency.
Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states.
Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended.
Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior.
edwinjosechittilappilly
force-pushed
the
fix-sharpoint
branch
from
May 19, 2026 17:25
01ec8b9 to
54cc34a
Compare
github-actions
Bot
added
bug
github-actions
Bot
added
bug
ricofurtado
pushed a commit
that referenced
this pull request
May 20, 2026
* Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Add *.db to .gitignore Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository. * Add orphan reconcile and bulk-delete helper Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely. - Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped). - Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input. - Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks. - Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior. Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures. * fix lint * style: ruff format (auto) * fix lint * Use max_files param and add typing Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency. * Add connector sync preview UI and API Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states. * style: ruff format (auto) * Hoist and consolidate imports in processors Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended. * Type UseQueryOptions with SearchResult Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
This was referenced May 20, 2026
Merged
ricofurtado
pushed a commit
that referenced
this pull request
May 22, 2026
* Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Add *.db to .gitignore Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository. * Add orphan reconcile and bulk-delete helper Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely. - Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped). - Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input. - Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks. - Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior. Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures. * fix lint * style: ruff format (auto) * fix lint * Use max_files param and add typing Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency. * Add connector sync preview UI and API Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states. * style: ruff format (auto) * Hoist and consolidate imports in processors Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended. * Type UseQueryOptions with SearchResult Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
ricofurtado
added a commit
that referenced
this pull request
May 22, 2026
…on (#1625) * feat: Implement in-process JWT claims caching with TTL and LRU eviction * style: ruff format (auto) * refactor: Update cache initialization to use settings for max size and TTL * feat: code cleanup * fix: Improve public key file reading and enhance token scheme handling * style: ruff autofix (auto) * fix: Handle user insert races and add test timeout (#1618) * Handle user insert races and add test timeout Increase TypeScript integration test timeout to 120s to reduce flakiness during slow CI runs. Enhance user_service.ensure_user_row IntegrityError handling to explicitly handle concurrent-insert races: detect a (oauth_provider, oauth_subject) race and return the existing row, detect an email_lookup_hash race by looking up the email and returning the concurrent identity when it matches, and handle PK collisions by retrying the insert with a new UUID. Add explanatory comments about which collisions are recoverable and when errors should propagate to the caller. * style: ruff format (auto) * Use PEP 604 union for agent config return type Replace typing.Optional[dict] with PEP 604 union syntax (dict | None) for get_effective_agent_config and remove the now-unused Optional import. This is a pure type-annotation cleanup (no runtime behavior changes); note it requires Python 3.10+ for the `|` union syntax. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: display file names and fix ingestion for onedrive (#1609) * fixed onedrive not working * style: ruff format (auto) * removed sites.read.all from onedrive * style: ruff format (auto) * fixed allowed users and groups * fix lint error * fixed lint * fixed mypy lint * style: ruff format (auto) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: Skip deleted connector files; return orphan IDs (#1592) * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Add *.db to .gitignore Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository. * Add orphan reconcile and bulk-delete helper Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely. - Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped). - Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input. - Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks. - Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior. Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures. * fix lint * style: ruff format (auto) * fix lint * Use max_files param and add typing Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency. * Add connector sync preview UI and API Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states. * style: ruff format (auto) * Hoist and consolidate imports in processors Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended. * Type UseQueryOptions with SearchResult Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: add reset db on command, expose error on e2e tests (#1627) * fix factory reset not deleting data * make onboarding errors pop up on e2e tests * style: ruff format (auto) * added check for error when uploading * added check if its on first step to rollback, not only if its complete * reset correctly * update timeout for uploading document * remove misclick * fixed lint * fix mypy errors * style: ruff format (auto) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * refactor: handler for file upload for context (#1624) * file upload for context * coderabbit suggestions * chore: add operator make commands (#1628) * Add kind build/load targets and kind sample Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters. * Add Makefile help for operator & kind Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root. * redirect on logout (#1440) (#1636) * fix: Copy flows directory into Docker image (#1632) Add a COPY instruction to include the local flows/ directory at /app/flows in the image so bundled flows are available at runtime. Placed before the entrypoint/ownership handling to ensure the files are copied with the intended UID/permissions. * feat: settings saas tabs menu (#1637) * remove logic for previous tab selected, need to be discussed with Ana * reduce repeated code and use darkmodelight class * reduce repeated code and use darkmodelight class * remove unecessary !important * remove unecessary !important * remove duplicate css blocks * remove !important and merge active and hover state * put back TabsContent anf fix hover issue --------- Co-authored-by: Olfa Maslah <olfamaslah@Olfas-MacBook-Pro.local> * chore: Add ruff autofix step to CI workflows (#1640) * Add ruff autofix step to CI workflows Add an in-place Ruff fix step to .github/workflows/autofix.ci.yml that runs uv run ruff check --fix on changed files, rename the job and autofix commit message to reflect "ruff autofix", and update comments to clarify that autofix.ci applies safe fixes and formatting. Also update .github/workflows/lint-backend.yml comments to state that safe lint fixes and formatting are auto-applied by autofix.ci while keeping strict lint (--no-fix) and mypy in the lint workflow. * Update autofix.ci.yml * fix: Enum IDs and delete OpenSearch docs by _id (#1638) * Enum IDs and delete OpenSearch docs by _id Add DLS-safe OpenSearch delete helpers and use them to avoid silent no-ops from delete_by_query. Introduces utils/opensearch_delete.py with collect_visible_document_ids and delete_document_ids (enumerate visible _ids via search/scroll, then delete by primary _id). Update delete_chunks_by_document_ids to enumerate chunk IDs and delete each by _id. Ensure langflow_connector_service and TaskProcessor clear stale chunks (by document_id) before re-ingest/re-index to prevent duplicate or trailing chunks after renames. Improve connector listing by scoping cfg.file_ids/folder_ids when available to avoid false orphan detection. Add and update unit tests to assert the new enumeration-and-delete behavior and ordering. * style: ruff format (auto) * ruff fix * ruff fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * make probes more robust (#1642) * feat: Show google_drive connector for cloud brand (#1650) Remove google_drive from the cloud-brand exclusion so Google Drive connectors are visible when isCloudBrand is true. Applied the change in connector-cards.tsx and knowledge-dropdown.tsx (OneDrive remains excluded). * operator ubi base build (#1652) * remove flow download initContainer when flowRef is removed (#1653) * fix: cr deletion (#1656) * fix cr deletion * refactoring * fix unit test * fix lint * fix cross namespace CR deletion * Potential fix for pull request finding 'CodeQL / Information exposure through an exception' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Edwin Jose <edwin.jose@datastax.com> Co-authored-by: Lucas Oliveira <62335616+lucaseduoli@users.noreply.github.com> Co-authored-by: Mike Fortman <mfortman11@gmail.com> Co-authored-by: Wallgau <46035189+Wallgau@users.noreply.github.com> Co-authored-by: Olfa Maslah <olfamaslah@Olfas-MacBook-Pro.local> Co-authored-by: ming <itestmycode@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
ricofurtado
pushed a commit
that referenced
this pull request
May 23, 2026
* Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Add *.db to .gitignore Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository. * Add orphan reconcile and bulk-delete helper Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely. - Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped). - Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input. - Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks. - Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior. Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures. * fix lint * style: ruff format (auto) * fix lint * Use max_files param and add typing Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency. * Add connector sync preview UI and API Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states. * style: ruff format (auto) * Hoist and consolidate imports in processors Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended. * Type UseQueryOptions with SearchResult Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
ricofurtado
added a commit
that referenced
this pull request
May 23, 2026
…on (#1625) * feat: Implement in-process JWT claims caching with TTL and LRU eviction * style: ruff format (auto) * refactor: Update cache initialization to use settings for max size and TTL * feat: code cleanup * fix: Improve public key file reading and enhance token scheme handling * style: ruff autofix (auto) * fix: Handle user insert races and add test timeout (#1618) * Handle user insert races and add test timeout Increase TypeScript integration test timeout to 120s to reduce flakiness during slow CI runs. Enhance user_service.ensure_user_row IntegrityError handling to explicitly handle concurrent-insert races: detect a (oauth_provider, oauth_subject) race and return the existing row, detect an email_lookup_hash race by looking up the email and returning the concurrent identity when it matches, and handle PK collisions by retrying the insert with a new UUID. Add explanatory comments about which collisions are recoverable and when errors should propagate to the caller. * style: ruff format (auto) * Use PEP 604 union for agent config return type Replace typing.Optional[dict] with PEP 604 union syntax (dict | None) for get_effective_agent_config and remove the now-unused Optional import. This is a pure type-annotation cleanup (no runtime behavior changes); note it requires Python 3.10+ for the `|` union syntax. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: display file names and fix ingestion for onedrive (#1609) * fixed onedrive not working * style: ruff format (auto) * removed sites.read.all from onedrive * style: ruff format (auto) * fixed allowed users and groups * fix lint error * fixed lint * fixed mypy lint * style: ruff format (auto) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: Skip deleted connector files; return orphan IDs (#1592) * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Skip deleted connector files; return orphan IDs Handle files deleted at source and improve orphan reconciliation. - Add TaskStatus.SKIPPED and update ConnectorFileProcessor to catch FileNotFoundError/ValueError from connector.get_file_content. When a file is missing, mark the file task as SKIPPED, set a skipped result, update timestamps and counters, and avoid raising so the upload can continue. - Change reconcile_orphans_for_connector_type to return a list of orphan file IDs ([]) instead of an int, update early returns and docstring to reflect returning the deleted/removed document IDs so callers can exclude them from subsequent sync passes. - Minor whitespace cleanup in sync_all_connectors. These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing. * Add *.db to .gitignore Add a '*.db' pattern to .gitignore to prevent local database files (e.g., SQLite) from being committed to the repository. * Add orphan reconcile and bulk-delete helper Introduce a reconcile pass and supporting bulk-delete utility to remove OpenSearch chunks for files that no longer exist remotely. - Add reconcile_orphans_for_connector_type (src/api/connectors.py): lists all active connections for a connector type, strictly gates on unauthenticated connections or listing errors (abort with 0 deletes), aggregates paginated remote file IDs, computes orphans (indexed IDs not present remotely) and invokes delete_chunks_by_document_ids. Integrated into connector_sync and sync_all_connectors (skips reconcile when sync is capped). - Add delete_chunks_by_document_ids (src/api/documents.py): issues a single delete_by_query with terms(document_id, ...) and conflicts="proceed", returns deleted count, and short-circuits on empty input. - Update connector metadata painless params (src/connectors/service.py): include filename param and assign ctx._source.filename = params.filename in the update script so renamed files update indexed chunks. - Add unit tests: cover delete_chunks_by_document_ids, reconcile_orphans_for_connector_type (gating, pagination, multi-connection union, error handling), and connector metadata filename behavior. Behavior notes: reconcile is conservative to avoid false-positive deletions; bulk delete is defensive and returns 0 on unexpected responses or failures. * fix lint * style: ruff format (auto) * fix lint * Use max_files param and add typing Add an explicit type annotation for files_to_process and update the connector.list_files call to use the max_files parameter instead of limit to match the connector API. Also remove redundant `# type: ignore` comments on connector.cfg assignments as they are no longer necessary. Minor cleanup to improve type clarity and API consistency. * Add connector sync preview UI and API Introduce sync-preview functionality for connectors and sync-all flows. Backend: add ID→filename aggregation, compute_orphans (non-destructive orphan detection), orphan deletion helper, and preview endpoints (connector_sync_preview, connectors_sync_all_preview) plus wiring in internal routes. Frontend: add useSyncConnectorPreview/useSyncAllConnectorsPreview mutations, wire preview flows into Knowledge pages/components, and add SyncConfirmDialog component to show orphan lists, per-connector synced counts, availability flags, and confirm deletion+sync interactions. Also update UI buttons to open the preview dialog and handle loading/syncing states. * style: ruff format (auto) * Hoist and consolidate imports in processors Move many inline imports to module scope and consolidate repeated imports across TaskProcessor and its subclasses. Adds top-level imports (asyncio, datetime, mimetypes, os, time), config/settings and utility functions (clients, get_embedding_model, get_index_name, get_openrag_config, extract_relevant, process_text_file, resplit_chunks_character_windows, ensure_embedding_field_exists, auto_cleanup_tempfile, hash_id, build_filename_delete_body, build_filename_search_body) and imports TaskStatus. Removes redundant per-function imports to improve readability and reduce repeated import overhead; no functional changes intended. * Type UseQueryOptions with SearchResult Specify UseQueryOptions<SearchResult> for the useGetSearchQuery hook's options parameter to improve TypeScript type inference and ensure the query options expect SearchResult data. This change tightens typings without altering runtime behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: add reset db on command, expose error on e2e tests (#1627) * fix factory reset not deleting data * make onboarding errors pop up on e2e tests * style: ruff format (auto) * added check for error when uploading * added check if its on first step to rollback, not only if its complete * reset correctly * update timeout for uploading document * remove misclick * fixed lint * fix mypy errors * style: ruff format (auto) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * refactor: handler for file upload for context (#1624) * file upload for context * coderabbit suggestions * chore: add operator make commands (#1628) * Add kind build/load targets and kind sample Add Makefile targets to build and load app images into a kind cluster (KIND_CLUSTER_NAME default: openrag): kind-load-app-images and kind-build-load-apps, and expose them in the help output. Update kubernetes/operator/README.md with instructions to run the operator locally, apply a kind-local sample, build/load images into kind, and restart deployments after rebuilding. Add a new config sample openrag_v1alpha1_openrag-kind-local.yaml tuned for 2-CPU kind/Colima clusters (lower resource requests, imagePullPolicy: Never). Also annotate the default sample to point users to the kind-local file for local clusters. * Add Makefile help for operator & kind Add a new help_operator target and OPERATOR_DIR variable to the Makefile, and include it in the .PHONY list. The new help output documents Kubernetes operator and kind-related commands (build/load app images into kind, operator deps/install/run/build/test/lint/manifests/generate, deploy/undeploy, docker-build, helm install) and provides a typical kind + local images workflow and sample CR usage. This makes it easier for developers to discover and run local operator/cluster tasks from the repo root. * redirect on logout (#1440) (#1636) * fix: Copy flows directory into Docker image (#1632) Add a COPY instruction to include the local flows/ directory at /app/flows in the image so bundled flows are available at runtime. Placed before the entrypoint/ownership handling to ensure the files are copied with the intended UID/permissions. * feat: settings saas tabs menu (#1637) * remove logic for previous tab selected, need to be discussed with Ana * reduce repeated code and use darkmodelight class * reduce repeated code and use darkmodelight class * remove unecessary !important * remove unecessary !important * remove duplicate css blocks * remove !important and merge active and hover state * put back TabsContent anf fix hover issue --------- Co-authored-by: Olfa Maslah <olfamaslah@Olfas-MacBook-Pro.local> * chore: Add ruff autofix step to CI workflows (#1640) * Add ruff autofix step to CI workflows Add an in-place Ruff fix step to .github/workflows/autofix.ci.yml that runs uv run ruff check --fix on changed files, rename the job and autofix commit message to reflect "ruff autofix", and update comments to clarify that autofix.ci applies safe fixes and formatting. Also update .github/workflows/lint-backend.yml comments to state that safe lint fixes and formatting are auto-applied by autofix.ci while keeping strict lint (--no-fix) and mypy in the lint workflow. * Update autofix.ci.yml * fix: Enum IDs and delete OpenSearch docs by _id (#1638) * Enum IDs and delete OpenSearch docs by _id Add DLS-safe OpenSearch delete helpers and use them to avoid silent no-ops from delete_by_query. Introduces utils/opensearch_delete.py with collect_visible_document_ids and delete_document_ids (enumerate visible _ids via search/scroll, then delete by primary _id). Update delete_chunks_by_document_ids to enumerate chunk IDs and delete each by _id. Ensure langflow_connector_service and TaskProcessor clear stale chunks (by document_id) before re-ingest/re-index to prevent duplicate or trailing chunks after renames. Improve connector listing by scoping cfg.file_ids/folder_ids when available to avoid false orphan detection. Add and update unit tests to assert the new enumeration-and-delete behavior and ordering. * style: ruff format (auto) * ruff fix * ruff fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * make probes more robust (#1642) * feat: Show google_drive connector for cloud brand (#1650) Remove google_drive from the cloud-brand exclusion so Google Drive connectors are visible when isCloudBrand is true. Applied the change in connector-cards.tsx and knowledge-dropdown.tsx (OneDrive remains excluded). * operator ubi base build (#1652) * remove flow download initContainer when flowRef is removed (#1653) * fix: cr deletion (#1656) * fix cr deletion * refactoring * fix unit test * fix lint * fix cross namespace CR deletion * Potential fix for pull request finding 'CodeQL / Information exposure through an exception' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Edwin Jose <edwin.jose@datastax.com> Co-authored-by: Lucas Oliveira <62335616+lucaseduoli@users.noreply.github.com> Co-authored-by: Mike Fortman <mfortman11@gmail.com> Co-authored-by: Wallgau <46035189+Wallgau@users.noreply.github.com> Co-authored-by: Olfa Maslah <olfamaslah@Olfas-MacBook-Pro.local> Co-authored-by: ming <itestmycode@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This was referenced Jun 1, 2026
This was referenced Jun 29, 2026
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Handle files deleted at source and improve orphan reconciliation.
These changes avoid aborting syncs on missing source files and make orphan deletions explicit for downstream processing.
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Chores