feat: Add custom metadata ingestion and filtering#2112
feat: Add custom metadata ingestion and filtering#2112edwinjosechittilappilly wants to merge 1 commit into
Conversation
github-actions
Bot
added
frontend
WalkthroughCustom metadata support is added end-to-end: fields are cataloged and validated, ingestion carries typed entries into indexed documents, search and chat accept structured metadata filters, frontend controls expose metadata editing, and Python, TypeScript, and MCP surfaces are updated. ChangesCustom metadata
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
React Doctor found 8 new issues in 3 files · 8 warnings · score 71 / 100 (Needs work) · 1 fixed · vs 8 warnings
Reviewed by React Doctor for commit |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| export function CustomMetadataEditor({ value, onChange }: Props) { | ||
| const [suggestions, setSuggestions] = useState<MetadataFieldSuggestion[]>([]); | ||
|
|
||
| useEffect(() => { |
There was a problem hiding this comment.
React Doctor · react-doctor/no-fetch-in-effect (warning)
fetch() inside useEffect can race, double-fire, or leak. Use a data-fetching layer or Server Component instead.
Fix → Use a data-fetching layer or Server Component so fetches do not race, double-fire, or leak from useEffect.
| return ( | ||
| <div | ||
| className="grid grid-cols-[minmax(0,1fr)_130px_minmax(0,1fr)_36px] gap-2" | ||
| key={`${index}-${entry.key}`} |
There was a problem hiding this comment.
React Doctor · react-doctor/no-array-index-as-key (warning)
Your users can see & submit the wrong data when this list reorders or filters, so use a stable id like key={item.id}, not the array index "index".
Fix → Use a stable id from the item, like key={item.id} or key={item.slug}. Index keys break when the list reorders or filters.
| for (const batch of batches) { | ||
| try { | ||
| const result = await uploadFiles(batch, replace); | ||
| const result = await uploadFiles( |
There was a problem hiding this comment.
React Doctor · react-doctor/async-await-in-loop (warning)
This makes the for…of loop slow because each await runs one after another, so collect the independent calls & run them together with await Promise.all(items.map(...))
Fix → Collect the items, then use await Promise.all(items.map(...)) so independent work runs at the same time
| const field = fields.find((candidate) => candidate.key === condition.key); | ||
| const valueListId = useId(); | ||
| const [suggestedValues, setSuggestedValues] = useState<unknown[]>([]); | ||
| useEffect(() => { |
There was a problem hiding this comment.
React Doctor · react-doctor/no-effect-event-handler (warning)
This useEffect is simulating an event handler, which costs an extra render & runs late.
Fix → Move event logic into the handler that starts it so the side effect does not run late after an extra render.
| const field = fields.find((candidate) => candidate.key === condition.key); | ||
| const valueListId = useId(); | ||
| const [suggestedValues, setSuggestedValues] = useState<unknown[]>([]); | ||
| useEffect(() => { |
There was a problem hiding this comment.
React Doctor · react-doctor/no-fetch-in-effect (warning)
fetch() inside useEffect can race, double-fire, or leak. Use a data-fetching layer or Server Component instead.
Fix → Use a data-fetching layer or Server Component so fetches do not race, double-fire, or leak from useEffect.
| fields={fields} | ||
| group={item} | ||
| // biome-ignore lint/suspicious/noArrayIndexKey: expression nodes have no persisted UI identifier | ||
| key={index} |
There was a problem hiding this comment.
React Doctor · react-doctor/no-array-index-as-key (warning)
Your users can see & submit the wrong data when this list reorders or filters, so use a stable id like key={item.id}, not the array index "index".
Fix → Use a stable id from the item, like key={item.id} or key={item.slug}. Index keys break when the list reorders or filters.
| condition={item} | ||
| fields={fields} | ||
| // biome-ignore lint/suspicious/noArrayIndexKey: expression nodes have no persisted UI identifier | ||
| key={index} |
There was a problem hiding this comment.
React Doctor · react-doctor/no-array-index-as-key (warning)
Your users can see & submit the wrong data when this list reorders or filters, so use a stable id like key={item.id}, not the array index "index".
Fix → Use a stable id from the item, like key={item.id} or key={item.slug}. Index keys break when the list reorders or filters.
| onChange: (value: MetadataGroup | undefined) => void; | ||
| }) { | ||
| const [fields, setFields] = useState<MetadataField[]>([]); | ||
| useEffect(() => { |
There was a problem hiding this comment.
React Doctor · react-doctor/no-fetch-in-effect (warning)
fetch() inside useEffect can race, double-fire, or leak. Use a data-fetching layer or Server Component instead.
Fix → Use a data-fetching layer or Server Component so fetches do not race, double-fire, or leak from useEffect.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| CustomMetadataService().normalize_entries(metadata_entries) | ||
| except ValueError as exc: | ||
| return JSONResponse({"error": f"Invalid metadata: {exc}"}, status_code=400) |
| await validate_metadata_query_data(normalized_query_data) | ||
| except Exception as e: | ||
| logger.error(f"Failed to normalize query_data: {e}") | ||
| return JSONResponse({"error": f"Invalid queryData format: {str(e)}"}, status_code=400) |
| CustomMetadataService().normalize_entries(parsed_metadata) | ||
| metadata_entries = parsed_metadata | ||
| except (json.JSONDecodeError, ValueError) as e: | ||
| return JSONResponse({"error": f"Invalid metadata: {e}"}, status_code=400) |
| try: | ||
| await CustomMetadataService().build_filter_clauses(resolved_filters) | ||
| except ValueError as exc: | ||
| return JSONResponse({"error": str(exc)}, status_code=400) |
| return JSONResponse({"results": results}) | ||
|
|
||
| except ValueError as e: | ||
| return JSONResponse({"error": str(e)}, status_code=400) |
github-actions
Bot
added
enhancement
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 18
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@alembic/versions/0008_metadata_fields.py`:
- Around line 25-26: Update the created_at and updated_at columns in
alembic/versions/0008_metadata_fields.py to use timezone-aware DateTime
definitions via timezone=True. Keep the existing datetime.now(UTC) behavior in
MetadataField and metadata_field_repo.py unchanged; those sibling sites require
no direct changes.
In `@frontend/components/cloud-picker/custom-metadata-editor.tsx`:
- Around line 75-97: Update the metadata row key in the custom metadata editor
to use a stable identifier that does not change when entry.key is edited,
preserving input focus across keystrokes. In the key input’s onChange handler,
normalize the entered key by removing characters rejected by backend validation,
including spaces and hyphens, before updating the entry and deriving its
suggested type.
In `@frontend/components/metadata-filter-builder.tsx`:
- Around line 128-134: Update the operator-change handler in the metadata
condition editor to reset the value to the shape required by the newly selected
operator, rather than retaining an incompatible prior value. Ensure equals and
other scalar operators receive scalar values, between receives a gte/lte range
object, and boolean in/not_in operators use an array-compatible value with an
appropriate list editor; otherwise remove those operators from the boolean
options. Apply the same value-shape handling to the related logic around the
second referenced section.
- Around line 66-67: Update scalarValue so an empty raw string is returned
unchanged before applying type-specific conversion, ensuring cleared numeric
fields and empty comma-separated entries do not become zero. Preserve the
existing number, boolean, and string conversions for non-empty values.
In `@src/api/custom_metadata.py`:
- Line 32: Instantiate CustomMetadataService in the application lifespan and
expose it through the dependency definitions, then inject that shared service
into _fields and _values in custom_metadata.py and into metadata validation in
knowledge_filter.py. Remove each local CustomMetadataService construction while
preserving the existing service calls; update all three sites:
src/api/custom_metadata.py lines 32 and 51-55, and src/api/knowledge_filter.py
lines 83-86.
- Around line 97-128: Define typed response models for field and value results,
then apply them to the public handlers list_fields, list_values, list_fields_v1,
and list_values_v1 through their return annotations or route response-model
configuration. Update _fields and _values as needed so each endpoint returns
model instances or compatible typed data instead of raw JSONResponse objects,
and verify FastAPI exposes the response schemas in OpenAPI.
In `@src/api/knowledge_filter.py`:
- Around line 127-132: Update the exception handling around normalize_query_data
and validate_metadata_query_data, including the corresponding block near the
second validation path, to catch the specific validation exception and return a
400 without exposing unexpected details. Handle all other exceptions separately
by logging the internal failure and returning a generic 500 response.
In `@src/api/schemas/custom_metadata.py`:
- Around line 10-35: Configure both recursive filter models,
MetadataFilterCondition and MetadataFilterGroup, to forbid unknown fields via
their Pydantic model configuration. Ensure nested condition and group payloads
reject extra or mixed properties instead of silently discarding them, while
preserving the existing fields and recursive structure.
In `@src/services/custom_metadata_service.py`:
- Around line 70-83: Remove the lazy database initialization fallback from
CustomMetadataService._session_factory and require a session factory via the
constructor. Instantiate CustomMetadataService with the configured factory in
main.py’s lifespan, then expose that instance through src/dependencies.py for
request handlers to consume rather than constructing the service directly.
- Around line 96-105: Update the registration loop in the custom metadata
normalization flow to use an insert-on-conflict/no-op operation instead of
separate get-then-add calls, making concurrent registration atomic. Then read
the stored record and compare its metadata type, preserving the intended
ValueError for conflicting types rather than exposing database uniqueness
errors. Add a test covering concurrent registration of the same key.
- Around line 217-251: Validate that the root expression and every group member
passed to _compile_node are dictionaries before calling .get() or recursively
traversing them. Raise the service’s validation ValueError for scalar nodes,
including scalar entries in conditions, while preserving existing compilation
behavior for valid object nodes.
- Around line 347-350: Update the number-validation branch in the custom
metadata normalization logic to reject non-finite or non-representable numeric
values, including NaN and positive or negative infinity, before appending to
normalized. Preserve existing bool and non-numeric rejection behavior, using the
appropriate numeric finiteness check around the validation in the metadata
handling method.
In `@src/services/document_index_writer.py`:
- Line 64: Use a single lifespan-managed CustomMetadataService instance: update
DocumentIndexWriter.__init__ in src/services/document_index_writer.py at lines
64-64 to accept and store the injected service instead of constructing it;
update both chat filter and nudge paths in src/services/chat_service.py at lines
158-160 and 259-261 to reuse that dependency; update SearchService.__init__ in
src/services/search_service.py at line 74 to accept it. Instantiate
CustomMetadataService in main.py’s lifespan block and inject the shared instance
through src/dependencies.py into all affected services and routers.
- Around line 315-324: Update the merge logic in the surrounding document-index
writing method to enforce the 50-field limit on the combined metadata,
validating merged_source after chunk and context metadata are merged and before
constructing the NormalizedMetadata return value. Reuse the existing metadata
validation mechanism and preserve the current merged-entry behavior.
- Around line 280-313: In the chunk metadata handling around custom metadata
extraction, copy the list assigned from the explicit metadata value before
extending it with arbitrary fields. Update the list branch in the code that
builds chunk_entries, preserving the existing handling for dictionary and other
metadata values while ensuring caller-owned chunk_metadata["metadata"] is not
mutated.
In `@src/services/langflow_file_service.py`:
- Around line 1069-1071: Validate metadata in
src/services/langflow_file_service.py at both the ingest-context construction
site (lines 1069-1071) and the related site at line 249, rejecting any non-list
value before it reaches DocumentIndexContext. Preserve valid list[dict[str,
Any]] values without coercing them, and update src/models/processors.py lines
1549-1555 so its metadata handling remains a plain pass-through for list-valued
settings with no validation or list conversion there.
In `@src/services/search_service.py`:
- Line 410: Update the _source allowlist in the search method containing
“custom_metadata” to also fetch the legacy “metadata” field, preserving the
existing custom_metadata selection and fallback at the source[“metadata”] access
around line 546.
In `@src/utils/opensearch_init.py`:
- Around line 228-235: Update the mapping initialization around
_ensure_field_mappings so metadata_entries is validated as an exact nested
mapping, not merely checked for field presence. Make mapping reconciliation
failures propagate and stop initialization before any metadata writes,
preventing ingestion from continuing when the field is absent or mapped as a
regular object.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: e3b9b8de-ad6f-421a-ba45-400d7bcd90b4
📒 Files selected for processing (50)
alembic/versions/0008_metadata_fields.pyfrontend/app/api/queries/useGetSearchQuery.tsfrontend/app/chat/_types/types.tsfrontend/components/cloud-picker/custom-metadata-editor.tsxfrontend/components/cloud-picker/ingest-settings.tsxfrontend/components/cloud-picker/types.tsfrontend/components/knowledge-dropdown.tsxfrontend/components/knowledge-filter-panel.tsxfrontend/components/metadata-filter-builder.tsxfrontend/contexts/knowledge-filter-context.tsxfrontend/lib/filter-normalization.tsfrontend/lib/upload-utils.tssdks/mcp/README.mdsdks/python/README.mdsdks/python/openrag_sdk/__init__.pysdks/python/openrag_sdk/documents.pysdks/python/openrag_sdk/models.pysdks/typescript/README.mdsdks/typescript/src/documents.tssdks/typescript/src/index.tssdks/typescript/src/types.tssrc/api/connectors.pysrc/api/custom_metadata.pysrc/api/knowledge_filter.pysrc/api/router.pysrc/api/schemas/custom_metadata.pysrc/api/v1/_filter_resolution.pysrc/api/v1/chat.pysrc/api/v1/documents.pysrc/api/v1/search.pysrc/app/routes/internal.pysrc/app/routes/public_v1.pysrc/config/settings.pysrc/db/models/__init__.pysrc/db/models/metadata_field.pysrc/db/repositories/__init__.pysrc/db/repositories/metadata_field_repo.pysrc/mcp_http/server.pysrc/models/processors.pysrc/services/chat_service.pysrc/services/custom_metadata_service.pysrc/services/document_index_writer.pysrc/services/langflow_file_service.pysrc/services/langflow_ingest_token_service.pysrc/services/search_service.pysrc/utils/embeddings.pysrc/utils/opensearch_init.pytests/unit/api/test_v1_filter_resolution.pytests/unit/services/test_custom_metadata_service.pytests/unit/test_langflow_ingest_callback.py
| sa.Column("created_at", sa.DateTime(), nullable=False), | ||
| sa.Column("updated_at", sa.DateTime(), nullable=False), |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify the production database backend and established timestamp convention.
rg -n -C3 'create_async_engine|postgres|sqlite|DATABASE' \
src/db/engine.py src/config/settings.py
rg -n -C2 'DateTime\(timezone=True\)|datetime\.now\(UTC\)' alembic srcRepository: langflow-ai/openrag
Length of output: 18104
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- alembic/versions/0008_metadata_fields.py ---'
sed -n '1,120p' alembic/versions/0008_metadata_fields.py
echo
echo '--- src/db/models/metadata_field.py ---'
sed -n '1,120p' src/db/models/metadata_field.py
echo
echo '--- src/db/repositories/metadata_field_repo.py ---'
sed -n '1,120p' src/db/repositories/metadata_field_repo.py
echo
echo '--- similar timestamp column definitions in neighboring models ---'
rg -n -C2 'sa\.DateTime\(|Field\(default_factory=lambda: datetime\.now\(UTC\)\)|sa_column=Column\(".*DateTime' src/db/models alembic/versions/0008_metadata_fields.pyRepository: langflow-ai/openrag
Length of output: 7103
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- metadata_field usages ---'
rg -n -C2 'MetadataField|created_at|updated_at' src | sed -n '1,220p'
echo
echo '--- timestamp handling patterns in repository/model code ---'
rg -n -C2 'datetime\.now\(UTC\)|isoformat\(\)|timezone=True' src/db src/api src/services alembic/versions | sed -n '1,260p'Repository: langflow-ai/openrag
Length of output: 33505
Align metadata_fields timestamps with UTC-aware columns.
alembic/versions/0008_metadata_fields.py defines created_at/updated_at as timezone-naive, but src/db/models/metadata_field.py and src/db/repositories/metadata_field_repo.py both use datetime.now(UTC). Set these columns to sa.DateTime(timezone=True) so the UTC offset is preserved in Postgres.
📍 Affects 3 files
alembic/versions/0008_metadata_fields.py#L25-L26(this comment)src/db/models/metadata_field.py#L13-L14src/db/repositories/metadata_field_repo.py#L23-L27
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@alembic/versions/0008_metadata_fields.py` around lines 25 - 26, Update the
created_at and updated_at columns in alembic/versions/0008_metadata_fields.py to
use timezone-aware DateTime definitions via timezone=True. Keep the existing
datetime.now(UTC) behavior in MetadataField and metadata_field_repo.py
unchanged; those sibling sites require no direct changes.
| <div | ||
| className="grid grid-cols-[minmax(0,1fr)_130px_minmax(0,1fr)_36px] gap-2" | ||
| key={`${index}-${entry.key}`} | ||
| > | ||
| <Input | ||
| aria-label="Metadata key" | ||
| list="custom-metadata-keys" | ||
| placeholder="supplier" | ||
| value={entry.key} | ||
| onChange={(event) => { | ||
| const key = event.target.value.toLowerCase(); | ||
| const suggestedType = fieldTypes.get(key); | ||
| update(index, { | ||
| key, | ||
| ...(suggestedType && suggestedType !== entry.type | ||
| ? { | ||
| type: suggestedType, | ||
| value: defaultValue(suggestedType), | ||
| } | ||
| : {}), | ||
| }); | ||
| }} | ||
| /> |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix React key to prevent typing focus loss.
Using entry.key in the React key prop forces the element to unmount and remount on every keystroke, causing the input to lose focus after the first character is typed.
Additionally, stripping invalid characters (like spaces or hyphens) during typing prevents users from encountering a 400 error on upload, keeping the input aligned with the backend validation rules.
🐛 Proposed fix
- <div
- className="grid grid-cols-[minmax(0,1fr)_130px_minmax(0,1fr)_36px] gap-2"
- key={`${index}-${entry.key}`}
- >
+ <div
+ className="grid grid-cols-[minmax(0,1fr)_130px_minmax(0,1fr)_36px] gap-2"
+ key={index}
+ >
<Input
aria-label="Metadata key"
list="custom-metadata-keys"
placeholder="supplier"
value={entry.key}
onChange={(event) => {
- const key = event.target.value.toLowerCase();
+ const key = event.target.value.toLowerCase().replace(/[^a-z0-9_]/g, "");
const suggestedType = fieldTypes.get(key);
update(index, {
key,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <div | |
| className="grid grid-cols-[minmax(0,1fr)_130px_minmax(0,1fr)_36px] gap-2" | |
| key={`${index}-${entry.key}`} | |
| > | |
| <Input | |
| aria-label="Metadata key" | |
| list="custom-metadata-keys" | |
| placeholder="supplier" | |
| value={entry.key} | |
| onChange={(event) => { | |
| const key = event.target.value.toLowerCase(); | |
| const suggestedType = fieldTypes.get(key); | |
| update(index, { | |
| key, | |
| ...(suggestedType && suggestedType !== entry.type | |
| ? { | |
| type: suggestedType, | |
| value: defaultValue(suggestedType), | |
| } | |
| : {}), | |
| }); | |
| }} | |
| /> | |
| <div | |
| className="grid grid-cols-[minmax(0,1fr)_130px_minmax(0,1fr)_36px] gap-2" | |
| key={index} | |
| > | |
| <Input | |
| aria-label="Metadata key" | |
| list="custom-metadata-keys" | |
| placeholder="supplier" | |
| value={entry.key} | |
| onChange={(event) => { | |
| const key = event.target.value.toLowerCase().replace(/[^a-z0-9_]/g, ""); | |
| const suggestedType = fieldTypes.get(key); | |
| update(index, { | |
| key, | |
| ...(suggestedType && suggestedType !== entry.type | |
| ? { | |
| type: suggestedType, | |
| value: defaultValue(suggestedType), | |
| } | |
| : {}), | |
| }); | |
| }} | |
| /> |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@frontend/components/cloud-picker/custom-metadata-editor.tsx` around lines 75
- 97, Update the metadata row key in the custom metadata editor to use a stable
identifier that does not change when entry.key is edited, preserving input focus
across keystrokes. In the key input’s onChange handler, normalize the entered
key by removing characters rejected by backend validation, including spaces and
hyphens, before updating the entry and deriving its suggested type.
| const scalarValue = (raw: string, type: MetadataField["type"] | undefined) => | ||
| type === "number" ? Number(raw) : type === "boolean" ? raw === "true" : raw; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve empty numeric input instead of converting it to zero.
Number("") is 0, so clearing a numeric field silently creates a zero filter; empty comma-separated entries also become zero. Handle the empty string before numeric conversion.
Proposed fix
-const scalarValue = (raw: string, type: MetadataField["type"] | undefined) =>
- type === "number" ? Number(raw) : type === "boolean" ? raw === "true" : raw;
+const scalarValue = (raw: string, type: MetadataField["type"] | undefined) => {
+ if (raw === "") return "";
+ if (type === "number") return Number(raw);
+ if (type === "boolean") return raw === "true";
+ return raw;
+};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const scalarValue = (raw: string, type: MetadataField["type"] | undefined) => | |
| type === "number" ? Number(raw) : type === "boolean" ? raw === "true" : raw; | |
| const scalarValue = (raw: string, type: MetadataField["type"] | undefined) => { | |
| if (raw === "") return ""; | |
| if (type === "number") return Number(raw); | |
| if (type === "boolean") return raw === "true"; | |
| return raw; | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@frontend/components/metadata-filter-builder.tsx` around lines 66 - 67, Update
scalarValue so an empty raw string is returned unchanged before applying
type-specific conversion, ensuring cleared numeric fields and empty
comma-separated entries do not become zero. Preserve the existing number,
boolean, and string conversions for non-empty values.
| onValueChange={(operator: MetadataOperator) => | ||
| onChange({ | ||
| ...condition, | ||
| operator, | ||
| value: | ||
| operator === "between" ? { gte: "", lte: "" } : condition.value, | ||
| }) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep each operator’s value shape consistent.
Switching from between to equals retains the range object, while boolean in/not_in renders and submits one boolean instead of an array. Reset values when operators change and provide a list editor—or omit list operators—for booleans.
Also applies to: 181-195
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@frontend/components/metadata-filter-builder.tsx` around lines 128 - 134,
Update the operator-change handler in the metadata condition editor to reset the
value to the shape required by the newly selected operator, rather than
retaining an incompatible prior value. Ensure equals and other scalar operators
receive scalar values, between receives a gte/lte range object, and boolean
in/not_in operators use an array-compatible value with an appropriate list
editor; otherwise remove those operators from the boolean options. Apply the
same value-shape handling to the related logic around the second referenced
section.
| }, | ||
| }, | ||
| ) | ||
| field_types = await CustomMetadataService().get_field_types() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Wire CustomMetadataService through lifespan and FastAPI dependency injection.
All three sites construct ad hoc service instances, potentially bypassing the configured session factory.
src/api/custom_metadata.py#L32-L32: inject the service into_fields.src/api/custom_metadata.py#L51-L55: inject the same service into_values.src/api/knowledge_filter.py#L83-L86: inject it into metadata validation rather than constructing it locally.
As per path instructions, “New services must be instantiated in the lifespan block in main.py and injected into routers via src/dependencies.py.”
📍 Affects 2 files
src/api/custom_metadata.py#L32-L32(this comment)src/api/custom_metadata.py#L51-L55src/api/knowledge_filter.py#L83-L86
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/api/custom_metadata.py` at line 32, Instantiate CustomMetadataService in
the application lifespan and expose it through the dependency definitions, then
inject that shared service into _fields and _values in custom_metadata.py and
into metadata validation in knowledge_filter.py. Remove each local
CustomMetadataService construction while preserving the existing service calls;
update all three sites: src/api/custom_metadata.py lines 32 and 51-55, and
src/api/knowledge_filter.py lines 83-86.
Source: Path instructions
| explicit = chunk_metadata.get("metadata") | ||
| if isinstance(explicit, list): | ||
| chunk_entries = explicit | ||
| elif isinstance(explicit, dict): | ||
| chunk_entries = self.custom_metadata.entries_from_mapping(explicit) | ||
| else: | ||
| chunk_entries = [] | ||
|
|
||
| known_chunk_fields = { | ||
| "allowed_groups", | ||
| "allowed_principal_labels", | ||
| "allowed_principals", | ||
| "allowed_users", | ||
| "chunk_overlap", | ||
| "chunk_size", | ||
| "connector_file_id", | ||
| "connector_type", | ||
| "created_time", | ||
| "document_id", | ||
| "file_size", | ||
| "filename", | ||
| "filesize", | ||
| "langflow_chunk_id", | ||
| "metadata", | ||
| "mimetype", | ||
| "modified_time", | ||
| "page", | ||
| "parser", | ||
| "source_url", | ||
| } | ||
| arbitrary = { | ||
| key: value for key, value in chunk_metadata.items() if key not in known_chunk_fields | ||
| } | ||
| chunk_entries.extend(self.custom_metadata.entries_from_mapping(arbitrary)) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Copy explicit entries before extending them.
chunk_entries = explicit aliases chunk.metadata["metadata"]; Line 313 then mutates caller-owned input. Rebuilding or retrying the same chunk can introduce duplicate entries and fail normalization.
Proposed fix
if isinstance(explicit, list):
- chunk_entries = explicit
+ chunk_entries = list(explicit)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| explicit = chunk_metadata.get("metadata") | |
| if isinstance(explicit, list): | |
| chunk_entries = explicit | |
| elif isinstance(explicit, dict): | |
| chunk_entries = self.custom_metadata.entries_from_mapping(explicit) | |
| else: | |
| chunk_entries = [] | |
| known_chunk_fields = { | |
| "allowed_groups", | |
| "allowed_principal_labels", | |
| "allowed_principals", | |
| "allowed_users", | |
| "chunk_overlap", | |
| "chunk_size", | |
| "connector_file_id", | |
| "connector_type", | |
| "created_time", | |
| "document_id", | |
| "file_size", | |
| "filename", | |
| "filesize", | |
| "langflow_chunk_id", | |
| "metadata", | |
| "mimetype", | |
| "modified_time", | |
| "page", | |
| "parser", | |
| "source_url", | |
| } | |
| arbitrary = { | |
| key: value for key, value in chunk_metadata.items() if key not in known_chunk_fields | |
| } | |
| chunk_entries.extend(self.custom_metadata.entries_from_mapping(arbitrary)) | |
| explicit = chunk_metadata.get("metadata") | |
| if isinstance(explicit, list): | |
| chunk_entries = list(explicit) | |
| elif isinstance(explicit, dict): | |
| chunk_entries = self.custom_metadata.entries_from_mapping(explicit) | |
| else: | |
| chunk_entries = [] | |
| known_chunk_fields = { | |
| "allowed_groups", | |
| "allowed_principal_labels", | |
| "allowed_principals", | |
| "allowed_users", | |
| "chunk_overlap", | |
| "chunk_size", | |
| "connector_file_id", | |
| "connector_type", | |
| "created_time", | |
| "document_id", | |
| "file_size", | |
| "filename", | |
| "filesize", | |
| "langflow_chunk_id", | |
| "metadata", | |
| "mimetype", | |
| "modified_time", | |
| "page", | |
| "parser", | |
| "source_url", | |
| } | |
| arbitrary = { | |
| key: value for key, value in chunk_metadata.items() if key not in known_chunk_fields | |
| } | |
| chunk_entries.extend(self.custom_metadata.entries_from_mapping(arbitrary)) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/services/document_index_writer.py` around lines 280 - 313, In the chunk
metadata handling around custom metadata extraction, copy the list assigned from
the explicit metadata value before extending it with arbitrary fields. Update
the list branch in the code that builds chunk_entries, preserving the existing
handling for dictionary and other metadata values while ensuring caller-owned
chunk_metadata["metadata"] is not mutated.
| chunk_normalized = self.custom_metadata.normalize_entries(chunk_entries) | ||
| context_normalized = self.custom_metadata.normalize_entries(context.metadata) | ||
| merged_source = {**chunk_normalized.source, **context_normalized.source} | ||
| merged_entries = {entry["key"]: entry for entry in chunk_normalized.index_entries} | ||
| merged_entries.update({entry["key"]: entry for entry in context_normalized.index_entries}) | ||
| from services.custom_metadata_service import NormalizedMetadata | ||
|
|
||
| return NormalizedMetadata( | ||
| source=merged_source, | ||
| index_entries=list(merged_entries.values()), |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Enforce the 50-field limit after merging.
Chunk and context entries are validated separately, so the final document can contain up to 100 fields despite the documented per-document limit. Validate merged_source before returning.
Proposed fix
merged_source = {**chunk_normalized.source, **context_normalized.source}
+if len(merged_source) > 50:
+ raise ValueError("A document cannot have more than 50 custom metadata fields")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| chunk_normalized = self.custom_metadata.normalize_entries(chunk_entries) | |
| context_normalized = self.custom_metadata.normalize_entries(context.metadata) | |
| merged_source = {**chunk_normalized.source, **context_normalized.source} | |
| merged_entries = {entry["key"]: entry for entry in chunk_normalized.index_entries} | |
| merged_entries.update({entry["key"]: entry for entry in context_normalized.index_entries}) | |
| from services.custom_metadata_service import NormalizedMetadata | |
| return NormalizedMetadata( | |
| source=merged_source, | |
| index_entries=list(merged_entries.values()), | |
| chunk_normalized = self.custom_metadata.normalize_entries(chunk_entries) | |
| context_normalized = self.custom_metadata.normalize_entries(context.metadata) | |
| merged_source = {**chunk_normalized.source, **context_normalized.source} | |
| if len(merged_source) > 50: | |
| raise ValueError("A document cannot have more than 50 custom metadata fields") | |
| merged_entries = {entry["key"]: entry for entry in chunk_normalized.index_entries} | |
| merged_entries.update({entry["key"]: entry for entry in context_normalized.index_entries}) | |
| from services.custom_metadata_service import NormalizedMetadata | |
| return NormalizedMetadata( | |
| source=merged_source, | |
| index_entries=list(merged_entries.values()), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/services/document_index_writer.py` around lines 315 - 324, Update the
merge logic in the surrounding document-index writing method to enforce the
50-field limit on the combined metadata, validating merged_source after chunk
and context metadata are merged and before constructing the NormalizedMetadata
return value. Reuse the existing metadata validation mechanism and preserve the
current merged-entry behavior.
| metadata=metadata | ||
| if metadata is not None | ||
| else (settings.get("metadata") if isinstance(settings, dict) else None), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== langflow_file_service.py relevant sections ==\n'
sed -n '220,280p' src/services/langflow_file_service.py
printf '\n---\n'
sed -n '1040,1095p' src/services/langflow_file_service.py
printf '\n== processors.py relevant section ==\n'
sed -n '1515,1575p' src/models/processors.py
printf '\n== search for metadata handling across repo ==\n'
rg -n "metadata\s*=" src/services src/models src -g '!**/__pycache__/**' | sed -n '1,200p'Repository: langflow-ai/openrag
Length of output: 16668
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== DocumentIndexContext definition ==\n'
rg -n "class DocumentIndexContext|metadata:" src -g '*.py' | sed -n '1,120p'
printf '\n== langflow_ingest_token_service.py relevant section ==\n'
sed -n '160,230p' src/services/langflow_ingest_token_service.py
printf '\n== document_index_writer.py relevant section ==\n'
sed -n '180,240p' src/services/document_index_writer.py
printf '\n== processors.py around metadata normalization ==\n'
sed -n '600,660p' src/models/processors.pyRepository: langflow-ai/openrag
Length of output: 10843
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== src/api/langflow_files.py ==\n'
sed -n '1,140p' src/api/langflow_files.py
printf '\n== src/api/langflow_ingest.py ==\n'
sed -n '1,120p' src/api/langflow_ingest.py
printf '\n== src/api/router.py metadata parsing ==\n'
sed -n '40,90p' src/api/router.py
printf '\n== search for upload_and_ingest_file callers ==\n'
rg -n "upload_and_ingest_file\(" src -g '*.py' | sed -n '1,120p'Repository: langflow-ai/openrag
Length of output: 12458
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== langflow_file_service.py around upload_and_ingest_file ==\n'
sed -n '880,980p' src/services/langflow_file_service.py
printf '\n== langflow_file_service.py around metadata forwarding ==\n'
sed -n '470,520p' src/services/langflow_file_service.py
printf '\n== connectors/service.py around upload_and_ingest_file caller ==\n'
sed -n '90,150p' src/connectors/service.py
printf '\n== processors.py around the other upload_and_ingest_file caller ==\n'
sed -n '1180,1235p' src/models/processors.pyRepository: langflow-ai/openrag
Length of output: 12483
Validate metadata before building the ingest context.
DocumentIndexContext.metadata is a list[dict[str, Any]]; list(metadata or []) will silently mangle non-list inputs (for example, a dict becomes a list of keys). Reject non-list values once in src/services/langflow_file_service.py, and keep src/models/processors.py as a plain pass-through for list-valued settings.
📍 Affects 2 files
src/services/langflow_file_service.py#L1069-L1071(this comment)src/services/langflow_file_service.py#L249-L249src/models/processors.py#L1549-L1555
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/services/langflow_file_service.py` around lines 1069 - 1071, Validate
metadata in src/services/langflow_file_service.py at both the ingest-context
construction site (lines 1069-1071) and the related site at line 249, rejecting
any non-list value before it reaches DocumentIndexContext. Preserve valid
list[dict[str, Any]] values without coercing them, and update
src/models/processors.py lines 1549-1555 so its metadata handling remains a
plain pass-through for list-valued settings with no validation or list
conversion there.
| "allowed_users", | ||
| "allowed_groups", | ||
| "allowed_principal_labels", | ||
| "custom_metadata", |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Fetch legacy metadata so the fallback can work.
Line 546 falls back to source["metadata"], but the _source allowlist only requests custom_metadata. Existing documents that only contain the legacy field therefore return empty metadata.
Proposed fix
"_source": [
...
"custom_metadata",
+ "metadata",
],Also applies to: 546-546
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/services/search_service.py` at line 410, Update the _source allowlist in
the search method containing “custom_metadata” to also fetch the legacy
“metadata” field, preserving the existing custom_metadata selection and fallback
at the source[“metadata”] access around line 546.
| await _ensure_field_mappings( | ||
| clients.opensearch, | ||
| index_name, | ||
| {"allowed_principal_labels": ACL_PRINCIPAL_LABELS_MAPPING}, | ||
| { | ||
| "allowed_principal_labels": ACL_PRINCIPAL_LABELS_MAPPING, | ||
| "custom_metadata": {"type": "object", "enabled": False}, | ||
| "metadata_entries": CUSTOM_METADATA_MAPPING, | ||
| }, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Do not treat the nested metadata mapping as best-effort.
The helper suppresses put_mapping failures and only checks field presence. If metadata_entries is absent or dynamically mapped as a regular object, ingestion can continue with an irreparable mapping while nested filters fail. Validate the exact type and stop initialization before metadata writes when reconciliation fails.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/utils/opensearch_init.py` around lines 228 - 235, Update the mapping
initialization around _ensure_field_mappings so metadata_entries is validated as
an exact nested mapping, not merely checked for field presence. Make mapping
reconciliation failures propagate and stop initialization before any metadata
writes, preventing ingestion from continuing when the field is absent or mapped
as a regular object.
2 participants
This pull request introduces support for custom metadata fields throughout the document ingest and search flow. It adds a new backend table for metadata fields, updates the ingest and search APIs and types to support typed custom metadata, and provides a UI for users to add and manage custom metadata during document upload. The changes ensure that metadata is consistently handled from ingestion to search and filtering.
Backend: Metadata Field Catalog
0008_metadata_fieldsto create a newmetadata_fieldstable for storing custom metadata field definitions, including their types.Type and API Updates
SearchPayload,SelectedFilters, and related types to include ametadatafield, supporting typed metadata for filtering and search. [1] [2] [3]ChunkResultto include ametadataproperty, enabling typed metadata to be returned in search results.metadatafield with typed entries.Frontend: Ingest UI and Flow
CustomMetadataEditorcomponent to allow users to add, edit, and remove typed custom metadata fields in the ingest UI.CustomMetadataEditorinto the advanced ingest settings UI, so users can specify metadata to apply to all uploaded documents. [1] [2]Frontend: Search and Filter Integration
MetadataFilterBuilderfor use in the knowledge filter panel, enabling future UI for filtering by custom metadata.These changes lay the groundwork for robust custom metadata support across document ingest, search, and filtering.
Summary by CodeRabbit
New Features
Documentation