Support a custom base URL for the OpenAI embedding/LLM provider (self-hosted gateway support)#2063
Open
galovics wants to merge 4 commits into
Open
Support a custom base URL for the OpenAI embedding/LLM provider (self-hosted gateway support)#2063galovics wants to merge 4 commits into
galovics wants to merge 4 commits into
Conversation
github-actions
Bot
added
backend
Author
Contributor
WalkthroughAdds OpenAI base URL handling across config loading, settings APIs, client construction, Langflow sync, and flow mapping. The PR also adds unit and integration tests covering persistence, validation, client wiring, and proxy request forwarding. ChangesOpenAI Base URL Override
Estimated code review effort: 3 (Moderate) | ~30 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/api/settings/endpoints.py (1)
660-691: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winClear
current_config.providers.openai.base_urlwhen removing OpenAI.
update_settingsonly updatesopenai_base_urlwhen it is explicitly sent, so leaving it here makes a later OpenAI re-enable keep routing through the old gateway instead of the default endpoint.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/api/settings/endpoints.py` around lines 660 - 691, The OpenAI removal path in update_settings leaves current_config.providers.openai.base_url behind, which can preserve an old gateway URL for later re-enables. In the remove_openai_config branch, alongside clearing api_key and configured on current_config.providers.openai, also reset base_url to an empty/default state so the next OpenAI enable uses the standard endpoint; use the existing update_settings and current_config.providers.openai symbols to place the change correctly.
🧹 Nitpick comments (1)
tests/unit/config/test_openai_base_url_config.py (1)
133-141: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUnrelated test in a base_url-specific test file.
test_normalize_model_name_still_distinguishes_cohere_variantstestsnormalize_model_namefromutils.embedding_fields, which has no connection to the OpenAIbase_urloverride. Including it intest_openai_base_url_config.pymuddies the file's purpose. Consider moving it to a more appropriate test module (e.g., atest_embedding_fields.pyor similar).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/config/test_openai_base_url_config.py` around lines 133 - 141, Move the unrelated normalize_model_name test out of the OpenAI base_url config test module and into a more appropriate embedding-focused test file. The test named test_normalize_model_name_still_distinguishes_cohere_variants imports normalize_model_name from utils.embedding_fields, so relocate it to a dedicated embedding test module (for example, a test_embedding_fields suite) and keep test_openai_base_url_config.py focused only on base_url behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/api/settings/models.py`:
- Line 29: The openai_base_url setting currently allows whitespace-only values
because Field(None, min_length=1) does not trim input, so update the validation
in the Settings model to strip leading/trailing whitespace before enforcing
non-empty input. Apply the fix in the openai_base_url field handling in
Settings, and make sure update_settings and onboarding still persist the
normalized value consistently so whitespace-only input is rejected instead of
becoming an empty override.
---
Outside diff comments:
In `@src/api/settings/endpoints.py`:
- Around line 660-691: The OpenAI removal path in update_settings leaves
current_config.providers.openai.base_url behind, which can preserve an old
gateway URL for later re-enables. In the remove_openai_config branch, alongside
clearing api_key and configured on current_config.providers.openai, also reset
base_url to an empty/default state so the next OpenAI enable uses the standard
endpoint; use the existing update_settings and current_config.providers.openai
symbols to place the change correctly.
---
Nitpick comments:
In `@tests/unit/config/test_openai_base_url_config.py`:
- Around line 133-141: Move the unrelated normalize_model_name test out of the
OpenAI base_url config test module and into a more appropriate embedding-focused
test file. The test named
test_normalize_model_name_still_distinguishes_cohere_variants imports
normalize_model_name from utils.embedding_fields, so relocate it to a dedicated
embedding test module (for example, a test_embedding_fields suite) and keep
test_openai_base_url_config.py focused only on base_url behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: cd1842f5-8744-44ad-9cea-78d43a08fb7d
📒 Files selected for processing (12)
.env.examplesrc/api/settings/endpoints.pysrc/api/settings/langflow_sync.pysrc/api/settings/models.pysrc/config/config_manager.pysrc/config/settings.pysrc/services/flows_service.pytests/integration/core/test_openai_custom_base_url_litellm_proxy.pytests/unit/config/test_openai_base_url_config.pytests/unit/config/test_patched_async_client_base_url.pytests/unit/services/test_flows_service_api_base_mapping.pytests/unit/test_settings_openai_base_url.py
galovics
added a commit
to galovics/openrag
that referenced
this pull request
Jul 9, 2026
- Clear providers.openai.base_url (not just api_key/configured) when removing the OpenAI config, so a later re-enable falls back to api.openai.com instead of silently reusing a stale gateway URL. - Reject whitespace-only openai_base_url in SettingsUpdateBody and OnboardingBody via a field_validator, instead of silently accepting it and stripping it down to an empty (no-op) value downstream. - Move test_normalize_model_name_still_distinguishes_cohere_variants out of the base_url-focused config test file into test_embedding_fields.py, where the rest of that module's coverage lives.
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/unit/test_settings_openai_base_url.py (1)
141-173: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider reusing
_patch_update_settings_depsin the onboarding test.The onboarding test duplicates the monkeypatch setup (lines 147-156) instead of calling
_patch_update_settings_deps. If the only reason is the additionalwait_for_langflowstub, you could extend the helper with an optional flag or add thewait_for_langflowpatch after calling it, reducing duplication.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/test_settings_openai_base_url.py` around lines 141 - 173, The onboarding test duplicates the dependency monkeypatch setup already covered by _patch_update_settings_deps; refactor test_onboarding_persists_openai_base_url to call that helper for the shared patches and then add only the extra wait_for_langflow stub needed for onboarding. If the helper lacks this hook, extend _patch_update_settings_deps with an optional flag or allow the caller to patch wait_for_langflow afterward, keeping the onboarding test focused on its unique behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/unit/test_settings_openai_base_url.py`:
- Around line 141-173: The onboarding test duplicates the dependency monkeypatch
setup already covered by _patch_update_settings_deps; refactor
test_onboarding_persists_openai_base_url to call that helper for the shared
patches and then add only the extra wait_for_langflow stub needed for
onboarding. If the helper lacks this hook, extend _patch_update_settings_deps
with an optional flag or allow the caller to patch wait_for_langflow afterward,
keeping the onboarding test focused on its unique behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 249faa4f-51c4-444e-87fe-d06983d0203c
📒 Files selected for processing (5)
src/api/settings/endpoints.pysrc/api/settings/models.pytests/unit/config/test_openai_base_url_config.pytests/unit/test_embedding_fields.pytests/unit/test_settings_openai_base_url.py
💤 Files with no reviewable changes (1)
- tests/unit/config/test_openai_base_url_config.py
🚧 Files skipped from review as they are similar to previous changes (2)
- src/api/settings/endpoints.py
- src/api/settings/models.py
…flow-ai#2060) Adds an openai.base_url override (config, env var OPENAI_BASE_URL, and settings API field) so OpenRAG's OpenAI provider can be pointed at a self-hosted OpenAI-compatible gateway (e.g. a LiteLLM proxy) instead of api.openai.com. Threads the override into all three AsyncOpenAI(...) construction sites in patched_async_client (including the HTTP/2 probe, which would otherwise still probe api.openai.com), and into the Langflow-flow ingest path via the existing api_base field-mapping / OPENAI_API_BASE global variable pattern already used for Ollama. Verified end to end against a local LiteLLM proxy + mock embeddings stub: the configured base_url reaches the real AsyncOpenAI client, and extra_body input_type passthrough works through the full chain.
SettingsUpdateBody.openai_base_url and OnboardingBody.openai_base_url were accepted by the settings/onboarding APIs but never applied to current_config.providers.openai.base_url, so a client POSTing openai_base_url got a silent no-op. Wire it up the same way watsonx_endpoint is applied in both handlers, but without marking OpenAI as "configured" on its own since base_url is an optional override, not a credential. Also add openai_base_url to the provider_fields list gating the providers:write RBAC check, matching the other non-secret endpoint fields (watsonx_endpoint, ollama_endpoint) already there.
- Clear providers.openai.base_url (not just api_key/configured) when removing the OpenAI config, so a later re-enable falls back to api.openai.com instead of silently reusing a stale gateway URL. - Reject whitespace-only openai_base_url in SettingsUpdateBody and OnboardingBody via a field_validator, instead of silently accepting it and stripping it down to an empty (no-op) value downstream. - Move test_normalize_model_name_still_distinguishes_cohere_variants out of the base_url-focused config test file into test_embedding_fields.py, where the rest of that module's coverage lives.
Upstream's atomicity fix (4d1be69) made update_settings() stage all mutations on a deep-copied working_config, only saving it at the end, instead of mutating the live current_config in place. The openai_base_url regression tests asserted on the original config object, which is no longer mutated - capture the object passed to save_config_file instead, matching the pattern used in test_settings_index_name_validation.py.
galovics
force-pushed
the
feat/openai-custom-base-url
branch
from
July 9, 2026 18:51
ea1b2fd to
5e20829
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/integration/core/test_openai_custom_base_url_litellm_proxy.py (1)
139-141: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueClean up
litellm.register_modelglobal state after the test.
litellm.register_modelmutates litellm's global model registry and is never reverted. If other tests in the same session use litellm with the same or overlapping model names, this could affect their provider classification. Consider snapshotting and restoring the registration in a fixture.♻️ Suggested cleanup fixture
+@pytest.fixture(autouse=True) +def _restore_litellm_model_registry(): + original = dict(litellm.model_cost) + yield + litellm.model_cost.clear() + litellm.model_cost.update(original) + + `@pytest.mark.skipif`( not _proxy_is_reachable(), reason=( "Local LiteLLM proxy not reachable at " f"{LITELLM_PROXY_URL} - skipping the real-proxy integration test." ), ) async def test_custom_base_url_reaches_real_litellm_proxy_with_input_type_passthrough(): # See module docstring: teaches litellm's local provider-detection table # that this proxy route alias belongs to "openai" so agentd's patched # client forwards the call to the real OpenAI SDK against our base_url # instead of raising BadRequestError before any network call happens. litellm.register_model( {EMBEDDING_MODEL: {"litellm_provider": "openai", "mode": "embedding"}} )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/core/test_openai_custom_base_url_litellm_proxy.py` around lines 139 - 141, The test mutates litellm’s global model registry via register_model and leaves that state behind. Update the test setup around the EMBEDDING_MODEL registration to snapshot the existing registry state before calling litellm.register_model, then restore it after the test (preferably with a fixture or teardown) so other tests are not affected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/integration/core/test_openai_custom_base_url_litellm_proxy.py`:
- Around line 139-141: The test mutates litellm’s global model registry via
register_model and leaves that state behind. Update the test setup around the
EMBEDDING_MODEL registration to snapshot the existing registry state before
calling litellm.register_model, then restore it after the test (preferably with
a fixture or teardown) so other tests are not affected.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 28e1e24f-4a1f-4d58-8107-b38ceaf3869f
📒 Files selected for processing (13)
.env.examplesrc/api/settings/endpoints.pysrc/api/settings/langflow_sync.pysrc/api/settings/models.pysrc/config/config_manager.pysrc/config/settings.pysrc/services/flows_service.pytests/integration/core/test_openai_custom_base_url_litellm_proxy.pytests/unit/config/test_openai_base_url_config.pytests/unit/config/test_patched_async_client_base_url.pytests/unit/services/test_flows_service_api_base_mapping.pytests/unit/test_embedding_fields.pytests/unit/test_settings_openai_base_url.py
✅ Files skipped from review due to trivial changes (2)
- src/api/settings/langflow_sync.py
- .env.example
🚧 Files skipped from review as they are similar to previous changes (9)
- tests/unit/services/test_flows_service_api_base_mapping.py
- src/config/config_manager.py
- tests/unit/test_embedding_fields.py
- src/api/settings/models.py
- src/config/settings.py
- src/services/flows_service.py
- tests/unit/test_settings_openai_base_url.py
- src/api/settings/endpoints.py
- tests/unit/config/test_patched_async_client_base_url.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2060
Summary
Adds an
openai.base_urloverride (config field,OPENAI_BASE_URLenv var, and settings API field) soOpenRAG's OpenAI provider can be pointed at a self-hosted OpenAI-compatible gateway (e.g. a LiteLLM proxy)
instead of
api.openai.com.AsyncOpenAI(...)construction sites inpatched_async_client(
src/config/settings.py), including the HTTP/2 capability probe — leaving that one unpatched wouldstill make a live call to
api.openai.comon every client init even with a custom gateway configured.field_mappings["api_base"]inflows_service.pypreviouslyonly mapped
ollama→OLLAMA_BASE_URL; this addsopenai→OPENAI_API_BASE, andlangflow_sync.pysets that Langflow global variable when the override is configured.
update_settingsand
onboardinghandlers, mirroring the existingwatsonx_endpointpattern — deliberately not markingopenai.configured = truefrom a barebase_url, since it's an optional override, not a credential, andconfiguredis consumed elsewhere as "this provider is a complete, usable setup."Why
OpenRAG's OpenAI provider always talks to
api.openai.comtoday. There's no way to route embedding/LLMtraffic through a centrally-managed self-hosted gateway without forking — a common pattern for teams running
their own LiteLLM proxy for unified cost tracking, virtual keys, or reaching providers OpenRAG doesn't
natively support (e.g. Cohere Embed Multilingual served via AWS Bedrock, reached through the gateway).
A correctness note for anyone extending this
OpenRAG's embedding client is patched by
agentd.patch.patch_openai_with_mcp. A model name with norecognized non-OpenAI litellm prefix resolves to
provider == "openai"and takes the real OpenAI SDK codepath (an actual HTTP call to whatever
base_urlis configured), not the in-process litellm SDK path. Thatmeans any non-standard parameter a downstream model needs (e.g. Cohere's
input_typewhen the gatewayfronts Cohere-on-Bedrock) must be passed via the OpenAI SDK's
extra_body, not as a bare kwarg, on thisspecific code path — verified against the real
agentdpatch source, not assumed.Testing
patched_async_clientconstruction(asserts the
base_urlkwarg directly across all three construction sites, with and without the overrideconfigured), the
flows_service.pyfield-mapping change, and the settings-API write path (persistscorrectly, doesn't mark
configured).override, builds the real
patched_async_client, calls.embeddings.create(..., extra_body={"input_type": "search_document"}), and confirms both a successful response and that the stub actually receivedinput_type="search_document"— proving the full chain (config → real OpenAI SDK → real Docker LiteLLMproxy → stub) works, including the
extra_bodypassthrough mechanism.uv run pytest tests/unit -q→ 901 passed, 13 failed. The 13 failures are pre-existing at thebase commit (unrelated to this change — collection errors from a partially-landed refactor upstream); zero
new failures.
Out of scope / follow-ups
litellm.register_model(...)call to teach litellm'slocal provider table that an arbitrary gateway route alias should resolve to
openai; production codedoes not replicate this. An arbitrary custom-gateway route name that litellm doesn't already recognize as
OpenAI (real model name, or
openai/-prefixed) will still raiseBadRequestErrorinagentdbefore anyHTTP call — a pre-existing
agentd/litellm constraint, not something this PR changes. Worth a trackedfollow-up if first-class support for arbitrary route aliases is wanted.
Summary by CodeRabbit
base_url(configured via setup/settings/onboarding) to route requests through OpenAI-compatible gateways.openai_base_urlis trimmed and rejects whitespace-only values.base_url..env.examplewithOPENAI_BASE_URL.