fix: add fixes to make e2e shard 8 be less prone to failures

[lucaseduoli]

This pull request makes several improvements to the Playwright test suite for model provider configuration, chat behavior, and URL ingestion. The main focus is on making tests more robust, expanding coverage for error cases, and improving reliability and error messaging.

Test robustness and reliability:

• Increased the timeout for chat row visibility in Chat.deleteChat to handle slower environments, improving test reliability.
• Modified chat response stability check to allow for shorter but stable responses, reducing false negatives in chat tests.

Model provider configuration and validation:

• Added explicit error handling and improved feedback for both Watsonx.ai and OpenAI provider configuration in Settings, including checking for success or error toasts and throwing descriptive errors on failure. [1] [2] [3]
• Updated z_update_model_providers.spec.ts to select specific language and embedding models after switching providers, and to skip tests if Watsonx credentials are not set. [1] [2] [3]

Test coverage and flexibility improvements:

• Expanded and generalized regular expressions in chat and URL ingestion tests to better match a variety of possible AI responses and error messages, making tests less brittle to LLM output changes. [1] [2] [3] [4]
• Added a verification step to ensure documents are active in the knowledge base before proceeding with chat or deletion, increasing test reliability for ingestion persistence tests.

Summary by CodeRabbit

• Tests
  • Broadened question matching for special-character input responses and prompt-injection/source-leak refusal phrasing.
  • Strengthened URL ingestion coverage with more flexible, case-insensitive failure/next-step checks, expanded access-denied keyword handling, and verification that ingested content becomes active before follow-up validation.
  • Improved chat streaming test robustness by refining stable-text detection and increasing the wait time before deleting chats.
  • Enhanced model-provider setup tests by skipping when Watsonx configuration is missing, selecting specific model settings during verification, and failing fast with clearer OpenAI connection/credential error handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 35cff7d0-fb5e-4a46-a604-d0d39514a558

📥 Commits

Reviewing files that changed from the base of the PR and between 16320eb and 19be625.

📒 Files selected for processing (1)

• frontend/tests/pages/Chat.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• frontend/tests/pages/Chat.ts

---

Walkthrough

Frontend Playwright tests broaden response assertions, configure model providers explicitly, synchronize document activation, normalize streamed tool calls, extend chat waits, and report Watsonx/OpenAI connection errors.

Changes

Frontend test reliability

Layer / File(s)	Summary
Broaden response and access assertions frontend/tests/core/basic-questions.spec.ts, frontend/tests/core/url_ingestion.spec.ts	Response regexes now cover broader refusal, failure, guidance, authentication, and access-restriction wording.
Synchronize provider and ingestion verification frontend/tests/core/z_update_model_providers.spec.ts, frontend/tests/core/url_ingestion.spec.ts	Watsonx-dependent tests skip when credentials are missing, configure selected Langflow models, and verify document activation before continuing.
Improve page-helper parsing and error handling frontend/tests/pages/Chat.ts, frontend/tests/pages/Settings.ts	Chat helpers normalize tool-call chunks and adjust waits, while provider configuration detects and reports connection errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: tests

Suggested reviewers: wallgau

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the PR's main goal of reducing e2e shard 8 failures and is broadly aligned with the test reliability fixes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/shard_8

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

frontend/tests/core/z_update_model_providers.spec.ts (1)

68-73: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Consider wrapping deleteDocument in a try...catch block to prevent test flakiness.

If the document does not exist (e.g., during a fresh test run or if a previous test failed before ingestion), deleteDocument may throw an error when attempting to locate the document row, which will fail the test. Wrapping it in a try...catch block, similar to how it is handled in url_ingestion.spec.ts, ensures the cleanup step is idempotent and resilient.

♻️ Proposed fix

     await settings.clickTab("Langflow");
     await settings.selectModel("Language model", "gpt-4o-mini");
     await settings.selectModel("Embedding model", "text-embedding-3-small");
-    await knowledge.deleteDocument(testDocumentName);
+    try {
+      await knowledge.deleteDocument(testDocumentName);
+    } catch (_error) {
+      // Document doesn't exist, safe to proceed
+    }
     await knowledge.ingestFile(testDocumentPath);
     await knowledge.verifyDocumentActive(testDocumentName);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@frontend/tests/core/z_update_model_providers.spec.ts` around lines 68 - 73,
Make the cleanup call to knowledge.deleteDocument(testDocumentName) in the test
setup resilient to a missing document by wrapping it in a try...catch, matching
the existing pattern in url_ingestion.spec.ts. Allow the test to continue when
deletion fails because the document is absent, then preserve the subsequent
ingestFile and verifyDocumentActive steps.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@frontend/tests/core/z_update_model_providers.spec.ts`:
- Around line 68-73: Make the cleanup call to
knowledge.deleteDocument(testDocumentName) in the test setup resilient to a
missing document by wrapping it in a try...catch, matching the existing pattern
in url_ingestion.spec.ts. Allow the test to continue when deletion fails because
the document is absent, then preserve the subsequent ingestFile and
verifyDocumentActive steps.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5727c775-c2bb-4d8c-b052-52570797030e

📥 Commits

Reviewing files that changed from the base of the PR and between 7fb88fd and 65f59ff.

📒 Files selected for processing (5)

• frontend/tests/core/basic-questions.spec.ts
• frontend/tests/core/url_ingestion.spec.ts
• frontend/tests/core/z_update_model_providers.spec.ts
• frontend/tests/pages/Chat.ts
• frontend/tests/pages/Settings.ts

[github-actions]

React Doctor found 1 new issue in 1 file · 1 warning · score 70 / 100 (Needs work) · 7 fixed · vs main

1 warning

app/chat/_components/chat-input.tsx

• ⚠️ L52 Large component is hard to read and change no-giant-component

_{Reviewed by React Doctor for commit b8de333. See inline comments for fixes.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@frontend/tests/pages/Chat.ts`:
- Around line 213-234: Update the streamed tool-call handling around
capturedToolData so entries from chunk.delta.tool_calls are accumulated across
chunks rather than locked on the first partial chunk. Preserve the initial tool
metadata, append each subsequent tc.function.arguments fragment to the existing
arguments, and continue processing until the complete streamed arguments are
available for downstream JSON.parse.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e923a2ec-fb99-47b2-b23c-52182da9813e

📥 Commits

Reviewing files that changed from the base of the PR and between 01dc583 and 16320eb.

📒 Files selected for processing (1)

• frontend/tests/pages/Chat.ts

[mpawlow]

Code Review 1

• ✅ Approved / LGTM 🚀