fix: cherrypick v1 updates from main to fix mcp and sdk (#1940)

* fix: Add filter_id support and v1 filter resolution (#1666)

* Add filter_id support and v1 filter resolution

Add server-side resolution for filter_id and extend document deletion to accept a filter_id. Introduces api.v1._filter_resolution to normalize a filter_id into concrete filters/limits/score_thresholds and to strip wildcards. v1 endpoints (chat, search, documents) now accept filter_id and merge resolved values with inline overrides; documents DELETE can delete all filenames referenced by a filter_id (with wildcard/empty data_sources rejected to avoid mass deletion). SDK updates (Python and TypeScript) allow DocumentsClient.delete to accept either filename or filter_id (mutually exclusive), include additional response fields (filenames, filter_id, per_file), and preserve idempotent semantics for filename deletes while surfacing filter-not-found errors. Tests updated/added to cover filter_id behavior, inline overrides, streaming, and validation. Additional minor logging/validation and type updates included.

* style: ruff autofix (auto)

* Clarify /v1/documents DELETE description

Expand the DELETE /v1/documents component description to indicate it can delete single or multiple documents and requires exactly one of `filename` or `filter_id`. Also notes that wildcards are rejected for safety, improving API documentation and removing ambiguity about deletion semantics.

* style: ruff autofix (auto)

* fix lint

* style: ruff autofix (auto)

* Set auth context for API-key search requests

Ensure API-key authenticated searches set the auth context and pass the user's JWT to the search service so downstream tools can resolve the user. Adds import for set_auth_context, calls it when handling API-key requests (mirroring v1 chat), and passes user.jwt_token to search_service.search instead of None. Also cleans up an unused typing import.

* Exclude ingest route; expose /v1 GETs as tools

Clarify search component docs to document `filter_id` and inline `filters` (inline filters override per-field). Document that /v1/documents/ingest is intentionally not customized/exposed via MCP because multipart/form-data uploads are not supported by FastMCP's from_fastapi conversion; clients should use the HTTP API or SDK to ingest. Add a RouteMap to explicitly exclude POST /v1/documents/ingest and consolidate route maps to expose all /v1/ routes (including GET) as MCP tools, with a note explaining that GETs are exposed as tools to better support LLM agent workflows.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
(cherry picked from commit eda6aaf)

* fix: more filter_id issues (#1730)

* missed filter_id fixes

* unit test

* clarify filter comments

* Use bracket notation for document body keys (#1734)

Change assignments to the request body to use bracket notation (body['filename'], body['filter_id']) instead of dot notation. This ensures the string keys are set explicitly on the Record<string, string> and avoids relying on declared property names.

* fix: Update integration.test.ts (#1737)

---------

Co-authored-by: Edwin Jose <edwin.jose@datastax.com>
(cherry picked from commit db9ae7f)

* Update documents.ts

* test: raise TS SDK filter-scope test timeouts and parallelize ingests

The "filterId in chat actually scopes retrieval to data_sources" integration
test was timing out at 60s and failing CI on every run. The chat itself works
(backend logs show the response generated in ~7.6s); the 60s budget was consumed
by two sequential Docling ingests (~25s each in CI) plus the filter create before
the chat even started, so the test crossed the deadline.

Bump the three ingest-heavy filter-scope tests (chat, search, delete-by-filter)
to a 180s per-test timeout and ingest the two docs concurrently via Promise.all
to shorten the critical path. Raise the global vitest testTimeout 60s -> 120s so
lighter tests have headroom too. The Python equivalent already passes because
pytest imposes no per-test cap; no backend/SDK behavior changes.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Sebastián Estévez <estevezsebastian@gmail.com>

Author: 3 people

sdks/python/openrag_sdk/documents.py

@@ -125,27 +125,46 @@ async def wait_for_task(
             await asyncio.sleep(poll_interval)
             elapsed += poll_interval
 
-        raise TimeoutError(f"Ingestion task {task_id} did not complete within {timeout}s")
+        raise TimeoutError(
+            f"Ingestion task {task_id} did not complete within {timeout}s"
+        )
 
-    async def delete(self, filename: str) -> DeleteDocumentResponse:
+    async def delete(
+        self,
+        filename: str | None = None,
+        *,
+        filter_id: str | None = None,
+    ) -> DeleteDocumentResponse:
         """
-        Delete a document from the knowledge base.
+        Delete document(s) from the knowledge base.
 
-        Args:
-            filename: Name of the file to delete.
+        Provide exactly one of:
+            filename: delete all chunks for that filename.
+            filter_id: delete chunks for each filename in the filter's data_sources.
 
         Returns:
             DeleteDocumentResponse with deleted chunk count.
         """
+        if bool(filename) == bool(filter_id):
+            raise ValueError("Provide exactly one of `filename` or `filter_id`")
+
+        body: dict[str, str] = {}
+        if filename is not None:
+            body["filename"] = filename
+        if filter_id is not None:
+            body["filter_id"] = filter_id
+
         try:
             response = await self._client._request(
                 "DELETE",
                 "/api/v1/documents",
-                json={"filename": filename},
+                json=body,
             )
         except NotFoundError as e:
-            # Keep delete idempotent for SDK callers: a missing document is not an exception.
-            if getattr(e, "status_code", None) == 404:
+            # Keep delete idempotent for SDK callers: a missing document is not
+            # an exception.
+            # Filter-not-found 404s still raise because the filter_id is caller input.
+            if filename is not None and getattr(e, "status_code", None) == 404:
                 return DeleteDocumentResponse(
                     success=False,
                     deleted_chunks=0,

sdks/python/openrag_sdk/models.py

@@ -1,7 +1,6 @@
 """OpenRAG SDK data models."""
 
-from datetime import datetime
-from typing import Any, Literal
+from typing import Literal
 
 from pydantic import BaseModel, Field
 
@@ -98,6 +97,10 @@ class DeleteDocumentResponse(BaseModel):
     filename: str | None = None
     message: str | None = None
     error: str | None = None
+    # Populated when deleting by filter_id — one entry per resolved data_source.
+    filenames: list[str] | None = None
+    filter_id: str | None = None
+    per_file: list[dict] | None = None
 
 
 # Chat history models

sdks/typescript/src/documents.ts

@@ -4,6 +4,7 @@
 
 import type { OpenRAGClient } from "./client";
 import type {
+  DeleteDocumentOptions,
   DeleteDocumentResponse,
   IngestResponse,
   IngestTaskStatus,
@@ -144,33 +145,57 @@ export class DocumentsClient {
   }
 
   /**
-   * Delete a document from the knowledge base.
+   * Delete document(s) from the knowledge base.
+   *
+   * Provide exactly one of:
+   *   - filename: a single filename, or
+   *   - { filename } / { filterId }: an options object.
    *
-   * @param filename - Name of the file to delete.
    * @returns DeleteDocumentResponse with deleted chunk count.
    */
-  async delete(filename: string): Promise<DeleteDocumentResponse> {
+  async delete(
+    arg: string | DeleteDocumentOptions
+  ): Promise<DeleteDocumentResponse> {
+    const opts: DeleteDocumentOptions =
+      typeof arg === "string" ? { filename: arg } : arg;
+
+    if (!opts.filename === !opts.filterId) {
+      throw new Error(
+        "Provide exactly one of `filename` or `filterId`"
+      );
+    }
+
+    const body: Record<string, string> = {};
+    if (opts.filename) body["filename"] = opts.filename;
+    if (opts.filterId) body["filter_id"] = opts.filterId;
+
     try {
       const response = await this.client._request("DELETE", "/api/v1/documents", {
-        body: JSON.stringify({ filename }),
+        body: JSON.stringify(body),
       });
 
       const data = await response.json();
       return {
         success: data.success ?? false,
         deleted_chunks: data.deleted_chunks ?? 0,
-        filename: data.filename ?? filename,
+        filename: data.filename ?? opts.filename ?? null,
         message: data.message ?? null,
         error: data.error ?? null,
+        filenames: data.filenames ?? null,
+        filter_id: data.filter_id ?? null,
+        per_file: data.per_file ?? null,
       };
     } catch (error) {
-      // Delete is idempotent: if no chunks match, backend may return 404.
-      // Surface this as a non-throwing "nothing deleted" response.
-      if ((error as NotFoundError)?.statusCode === 404) {
+      // Filename delete stays idempotent (404 -> success:false). Filter-id 404s
+      // are caller errors (bad filter id) and should propagate.
+      if (
+        opts.filename &&
+        (error as NotFoundError)?.statusCode === 404
+      ) {
         return {
           success: false,
           deleted_chunks: 0,
-          filename,
+          filename: opts.filename,
           message: null,
           error: (error as Error)?.message ?? "Resource not found",
         };

sdks/typescript/src/types.ts

@@ -77,6 +77,15 @@ export interface DeleteDocumentResponse {
   filename?: string | null;
   message?: string | null;
   error?: string | null;
+  // Populated when deleting by filter_id — one entry per resolved data_source.
+  filenames?: string[] | null;
+  filter_id?: string | null;
+  per_file?: Array<Record<string, unknown>> | null;
+}
+
+export interface DeleteDocumentOptions {
+  filename?: string;
+  filterId?: string;
 }
 
 // Chat history types

sdks/typescript/tests/integration.test.ts

@@ -190,55 +190,153 @@ describe.skipIf(SKIP_TESTS)("OpenRAG TypeScript SDK Integration", () => {
       expect(filter).toBeNull();
     });
 
-    it("should use filterId in chat", async () => {
-      // Create a filter first
+    it("filterId in chat actually scopes retrieval to data_sources", async () => {
+      // Ingest two distinguishable docs.
+      const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "sdk-filter-"));
+      const alphaName = `alpha_${Date.now()}.md`;
+      const betaName = `beta_${Date.now()}.md`;
+      const alphaPath = path.join(tmpDir, alphaName);
+      const betaPath = path.join(tmpDir, betaName);
+      fs.writeFileSync(alphaPath, "# Alpha\n\nPurple elephants live here.\n");
+      fs.writeFileSync(betaPath, "# Beta\n\nYellow tigers live here.\n");
+      await Promise.all([
+        client.documents.ingest({ filePath: alphaPath }),
+        client.documents.ingest({ filePath: betaPath }),
+      ]);
+
       const createResult = await client.knowledgeFilters.create({
-        name: "Chat Test Filter",
-        description: "Filter for testing chat with filterId",
+        name: `TS chat filter scope ${Date.now()}`,
+        description: "Filter scoped to alpha only",
         queryData: {
-          query: "test",
-          limit: 5,
+          query: "",
+          filters: {
+            data_sources: [alphaName],
+            document_types: ["*"],
+            owners: ["*"],
+            connector_types: ["*"],
+          },
+          limit: 10,
+          scoreThreshold: 0,
         },
       });
       expect(createResult.success).toBe(true);
       const filterId = createResult.id!;
 
       try {
-        // Use filter in chat
         const response = await client.chat.create({
-          message: "Hello with filter",
+          message: "What animals appear in these documents?",
           filterId,
         });
+        expect(response.sources).toBeDefined();
+        const names = (response.sources ?? []).map((s) => s.filename);
+        // Beta must NOT leak through the filter.
+        expect(names).not.toContain(betaName);
+      } finally {
+        await client.knowledgeFilters.delete(filterId);
+        await client.documents.delete(alphaName);
+        await client.documents.delete(betaName);
+      }
+    }, 180_000);
+
+    it("filterId in search actually scopes results to data_sources", async () => {
+      const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "sdk-filter-"));
+      const alphaName = `alpha_${Date.now()}.md`;
+      const betaName = `beta_${Date.now()}.md`;
+      const alphaPath = path.join(tmpDir, alphaName);
+      const betaPath = path.join(tmpDir, betaName);
+      fs.writeFileSync(alphaPath, "# Alpha\n\nPurple elephants live here.\n");
+      fs.writeFileSync(betaPath, "# Beta\n\nYellow tigers live here.\n");
+      await Promise.all([
+        client.documents.ingest({ filePath: alphaPath }),
+        client.documents.ingest({ filePath: betaPath }),
+      ]);
 
-        expect(response.response).toBeDefined();
+      const createResult = await client.knowledgeFilters.create({
+        name: `TS search filter scope ${Date.now()}`,
+        description: "Filter scoped to alpha only",
+        queryData: {
+          query: "",
+          filters: {
+            data_sources: [alphaName],
+            document_types: ["*"],
+            owners: ["*"],
+            connector_types: ["*"],
+          },
+          limit: 10,
+          scoreThreshold: 0,
+        },
+      });
+      expect(createResult.success).toBe(true);
+      const filterId = createResult.id!;
+
+      try {
+        const results = await client.search.query("animals", { filterId });
+        for (const r of results.results) {
+          expect(r.filename).not.toBe(betaName);
+        }
       } finally {
-        // Cleanup
         await client.knowledgeFilters.delete(filterId);
+        await client.documents.delete(alphaName);
+        await client.documents.delete(betaName);
       }
-    });
+    }, 180_000);
+
+    it("documents.delete(filterId) only removes filenames in the filter", async () => {
+      const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "sdk-filter-"));
+      const alphaName = `alpha_${Date.now()}.md`;
+      const betaName = `beta_${Date.now()}.md`;
+      const alphaPath = path.join(tmpDir, alphaName);
+      const betaPath = path.join(tmpDir, betaName);
+      fs.writeFileSync(alphaPath, "# Alpha\n\nPurple elephants.\n");
+      fs.writeFileSync(betaPath, "# Beta\n\nYellow tigers.\n");
+      await Promise.all([
+        client.documents.ingest({ filePath: alphaPath }),
+        client.documents.ingest({ filePath: betaPath }),
+      ]);
 
-    it("should use filterId in search", async () => {
-      // Create a filter first
       const createResult = await client.knowledgeFilters.create({
-        name: "Search Test Filter",
-        description: "Filter for testing search with filterId",
+        name: `TS delete-by-filter ${Date.now()}`,
+        description: "Filter scoped to alpha only",
         queryData: {
-          query: "test",
-          limit: 5,
+          query: "",
+          filters: {
+            data_sources: [alphaName],
+            document_types: ["*"],
+            owners: ["*"],
+            connector_types: ["*"],
+          },
+          limit: 10,
+          scoreThreshold: 0,
         },
       });
       expect(createResult.success).toBe(true);
       const filterId = createResult.id!;
 
       try {
-        // Use filter in search
-        const results = await client.search.query("test query", { filterId });
+        const result = await client.documents.delete({ filterId });
+        expect(result.success).toBe(true);
+        expect(result.filenames).toContain(alphaName);
+        expect(result.filenames ?? []).not.toContain(betaName);
 
-        expect(results.results).toBeDefined();
+        // Beta still searchable
+        const remaining = await client.search.query("tigers");
+        const remainingNames = remaining.results.map((r) => r.filename);
+        expect(remainingNames).toContain(betaName);
       } finally {
-        // Cleanup
         await client.knowledgeFilters.delete(filterId);
+        await client.documents.delete(alphaName);
+        await client.documents.delete(betaName);
       }
+    }, 180_000);
+
+    it("documents.delete rejects both filename and filterId together", async () => {
+      await expect(
+        client.documents.delete({ filename: "x.pdf", filterId: "y" })
+      ).rejects.toThrow();
+    });
+
+    it("documents.delete rejects when neither arg is set", async () => {
+      await expect(client.documents.delete({})).rejects.toThrow();
     });
   });
 

sdks/typescript/vitest.config.ts

@@ -3,7 +3,7 @@ import { defineConfig } from "vitest/config";
 export default defineConfig({
   test: {
     include: ["tests/**/*.test.ts"],
-    testTimeout: 60000, // 60 second timeout for integration tests
+    testTimeout: 120000, // 120 second timeout for integration tests
     hookTimeout: 60000,
   },
 });