fix: keep default docs title the same for every type of ingestion

[lucaseduoli]

This pull request enhances the "Fetch Latest Docs" feature by improving end-to-end test coverage for both Langflow-enabled and traditional ingestion modes, and refactors backend ingestion logic to better handle document metadata. The main changes include updating tests to toggle Langflow ingestion, modifying ingestion services to use document titles from URLs, and improving type safety for optional filenames throughout the codebase.

Testing improvements:

• The E2E test in fetch_latest_docs.spec.ts now verifies "Fetch Latest Docs" works both with Langflow ingestion enabled and disabled, toggling the setting and validating the active document in each mode.
• The Settings test page object adds a method to toggle Langflow ingestion and locates the relevant UI element for the test. [1] [2]

Backend ingestion and metadata handling:

• materialize_url_as_text_file now returns both the temporary file path and the document title, allowing ingestion services to use the actual title as the original_filename instead of a hardcoded value. [1] [2] [3] [4]
• The filename parameter in document ingestion contexts and callbacks is now optional (str | None), improving type safety and flexibility for documents without explicit filenames. [1] [2] [3]

Summary by CodeRabbit

• Improvements

  • Default documents now use the source page title, making them easier to identify.
  • URL-based document ingestion now handles missing filenames more reliably.

• Bug Fixes

  • Fetching the latest default documentation is now consistent whether Langflow ingestion is enabled or disabled.
  • Default documentation verification recognizes the correct document title.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f4a5d834-71ed-4742-90c7-12d5657a48ca

📥 Commits

Reviewing files that changed from the base of the PR and between 787ea27 and 4f3408a.

📒 Files selected for processing (2)

• frontend/tests/core/fetch_latest_docs.spec.ts
• src/utils/url_content_fetcher.py

🚧 Files skipped from review as they are similar to previous changes (2)

• frontend/tests/core/fetch_latest_docs.spec.ts
• src/utils/url_content_fetcher.py

---

Walkthrough

The PR propagates page titles into default-document metadata, makes URL ingestion filenames optional, and expands E2E coverage to validate fetching with Langflow ingestion disabled and enabled.

Changes

Default document ingestion

Layer / File(s)	Summary
Propagate materialized page titles src/utils/url_content_fetcher.py, src/services/default_docs_service.py	URL materialization returns a temporary file path and page title; default-document processing uses the title as original_filename, with OpenRAG as the fallback title.
Allow optional URL ingest filenames src/services/document_index_writer.py, src/services/langflow_file_service.py	filename is optional in ingestion context and callbacks, and URL ingestion no longer passes the URL as the filename.
Exercise both ingestion modes frontend/tests/pages/Settings.ts, frontend/tests/core/fetch_latest_docs.spec.ts	The test toggles Langflow ingestion off and on, fetches latest docs in both modes, and verifies the expected default document remains active.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant FetchLatestDocsTest
  participant Settings
  participant KnowledgePage
  FetchLatestDocsTest->>Settings: Disable Langflow ingestion
  FetchLatestDocsTest->>KnowledgePage: Fetch latest docs
  KnowledgePage-->>FetchLatestDocsTest: Verify default document
  FetchLatestDocsTest->>Settings: Enable Langflow ingestion
  FetchLatestDocsTest->>KnowledgePage: Fetch latest docs
  KnowledgePage-->>FetchLatestDocsTest: Verify default document

Loading

Possibly related PRs

• langflow-ai/openrag#1982: Changes filename propagation during URL ingestion in langflow_file_service.py.

Suggested labels: bug, tests

Suggested reviewers: mfortman11, edwinjosechittilappilly

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: consistent default docs title handling across ingestion modes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/default_document_title

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mpawlow]

Code Review 1

• ✅ Approved / LGTM 🚀
• See Minor PR comments: (1a), (1c) for optional consideration

[mpawlow]

(1a) [Minor] E2E test timeout not increased despite doubling the test's workload

Problem

• The test was extended from a single ingestion pass to two full sequential phases (Langflow-disabled, then Langflow-enabled), each of which:
  • Toggles a setting and waits for a save-confirmation toast (up to 120s per saveIngestSettings() call).
  • Triggers fetchLatestDocs().
  • Polls verifyDocumentActive(), which has its own internal 5-minute (300000ms) deadline waiting for OpenSearch indexing to complete.
• Despite the workload roughly doubling, test.setTimeout(180000) (3 minutes) was left unchanged.
• Even a single call to verifyDocumentActive() can internally poll for up to 5 minutes, which already exceeds the current 3-minute outer test timeout — this pre-existing mismatch is now compounded by running two full phases back-to-back.

Code References

• frontend/tests/core/fetch_latest_docs.spec.ts:11 (test.setTimeout(180000), unchanged)
• frontend/tests/core/fetch_latest_docs.spec.ts:20-46 (two sequential phases added)
• frontend/tests/pages/Settings.ts:197-203 (saveIngestSettings(), up to 120s wait)
• frontend/tests/pages/Knowledge.ts:717 (verifyDocumentActive(), 300000ms internal deadline)

Potential Solution

• Increase test.setTimeout(...) to comfortably cover two full phases, e.g. 360000 (6 minutes) or more, matching realistic worst-case indexing + save-toast latency for both phases combined.

Alternative Solutions

• Split into two separate test(...) cases (Langflow-disabled, Langflow-enabled) each with its own 3-minute budget, so a slow phase doesn't consume budget needed by the other and failures are individually attributable.

[mpawlow]

(1c) [Minor] Empty <title></title> produces an empty original_filename

Problem

• materialize_url_as_text_file falls back to "OpenRAG" only when the <title> tag is entirely absent (title_match is None).
• If the tag is present but empty (<title></title>) or whitespace-only, title_match.group(1).strip() evaluates to "", and title = "" is used verbatim — no fallback is applied.
• This empty string then becomes original_filename="" in _ingest_default_documents_url, producing a default document with an empty display name.

Code References

• src/utils/url_content_fetcher.py:51
• src/services/default_docs_service.py:344 (original_filename=title)

Potential Solution

• Fall back to "OpenRAG" when the extracted title is falsy after stripping, not only when the regex fails to match:

  title = html.unescape(title_match.group(1).strip()) if title_match else ""
  title = title or "OpenRAG"