⚡ Bolt: [performance improvement] VectorStore O(N^2) insertion bottleneck fix#48
Closed
sl4m3 wants to merge 348 commits into
Closed
⚡ Bolt: [performance improvement] VectorStore O(N^2) insertion bottleneck fix#48sl4m3 wants to merge 348 commits into
sl4m3 wants to merge 348 commits into
Conversation
- Activate autonomous heartbeat loop in MCPServer with persistent timers - Implement physical vector deletion in VectorStore upon memory forget - Enable persistent Git indexing via SemanticMetaStore configuration - Integrate ConflictEngine and TransitionValidator for deep integrity - Implement evidence-based ranking and popularity tracking (hit_count) - Enhance KnowledgeGraph with evidence counts and Mermaid links - Integrate TransactionManager for synchronized file/DB rollbacks - Activate REST Gateway endpoints, structured JSON logging, and API spec - Update project version to 2.5.1 and add comprehensive verification suite
…5.2) - Fix FileSystemLock bug preventing correct fcntl lock wait/retry - Implement late-bind conflict detection inside transactions to prevent race conditions - Index decision rationale in VectorStore and SemanticMetaStore for improved search relevance - Update IntegrationBridge to return structured context with knowledge markers - Bump version to 2.5.2 (core, package, and MCP API)
- Update README.md to reflect v2.5.2 features and structure - Translate INTEGRATION_GUIDE.md, QUICKSTART.md, and COMPARISON.md to Russian - Refine documentation style to match project standards (BLUF, 80-char wrap, active voice) - Fix section numbering and improve navigation links in tutorials
- Implement Metadata Cache for full context in SemanticMetaStore (context_json) - Refactor ReflectionEngine to eliminate N+1 file reads by using Metadata Cache - Implement Singleton pattern for MCPServer maintenance thread with threading.Lock - Add multi-process safety to maintenance loop using non-blocking FileSystemLock - Fix TypeError in FileSystemLock.acquire by adding timeout support - Update test suite to handle singleton server constraints and verify concurrency - Bump project documentation and ensure Russian localization consistency
- Update version to 2.5.3 in pyproject.toml, README, and core modules - Optimize ReflectionEngine: switch to metadata-based target filtering - Enhance SemanticStore: improve file walking and transaction safety - Improve test suite: add ML mocking and optimized storage fixtures - Fix: refine decay logic to protect permanent decisions
…conflict resolution (v2.6.0)
…tHub compatibility
…cise pixel-perfect alignment
…elisted SQL injection warning, and fix Heartbeat test for incremental watermark
…e bridge API - Added turn-based memory retention and deduplication in IntegrationBridge to prevent redundant context injection. - Enhanced bridge API with proxy methods for core memory operations (record, supersede, search, etc.). - Fixed context injection to include memory content/rationale for better agent grounding. - Updated tests to cover extended bridge functionality.
- Updated version in pyproject.toml and README.md. - Refreshed benchmarks section with current performance data from Termux. - Documented new context deduplication feature.
…ents - Standardized on Jina v5 Nano for high-precision semantic search. - Implemented advanced knowledge grounding: link evidence to semantic records during evolution. - Enhanced ReflectionEngine with automatic evidence aggregation and falsification logic. - Improved system resilience with automatic database recovery (Self-Healing). - Added multi-step truth resolution for long evolution chains. - Fixed resource management: ensure proper database closure in tests and production. - Refreshed benchmarks at 1000/5000 scale. - Updated documentation and README to reflect 2.7.0 capabilities.
…est data - Switched SemanticMetaStore to 'isolation_level=None' (autocommit) to allow manual SAVEPOINTs from TransactionManager to coexist with standalone operations. - Ensured all standalone database writes in SemanticMetaStore are committed immediately to prevent 'database is locked' errors in concurrent environments. - Wrapped 'Memory.update_decision' and 'Memory.run_decay' in transactions for atomicity and consistency. - Updated 'TestDeepIntegrity.test_conflict_prevention' to use radically different content, ensuring it triggers a 'CONFLICT' instead of the new 'Auto-Supersede' behavior. - Verified all tests pass locally with concurrency (pytest -n4).
- Implemented unified error handling with Result pattern. - Added SQLAlchemy connection pooling for episodic storage. - Integrated comprehensive health check endpoints. - Hardened input sanitization for memory events. - Updated documentation and version to 3.1.2.
- Extended ProposalContent in schemas.py with phase, vitality, stability_score, and frequency. - Updated ReflectionEngine to inject decision_id into proposals, allowing them to be processed by LifecycleEngine as DecisionStreams. - Fixed old tests that failed to unpack the new Result type from EpisodicStore. - Procedural knowledge (instructions) now properly evolves from PATTERN to EMERGENT and decays if unused.
- Updated bridge-record CLI to support reading model responses from stdin using the '-' flag. - Refactored Claude and Cursor installers to use pipes instead of command-line arguments for responses. - Eliminates shell escaping issues and 'argument list too long' errors for large agent responses. - Fixed record ID 5277-like truncation issues in episodic memory.
Previous attempt failed to actually remove the lines. This check was blocking legitimate Russian/English content.
- Large agent responses and complex tool traces were exceeding the previous 100KB limit. - Increased threshold to 500KB to ensure reliable interaction recording in high-context scenarios. - Maintained DoS protection by keeping a reasonable upper bound.
…cal logs - Removed heuristic checks for code injection and Windows paths that caused false positives. - These checks blocked legitimate recording of tool calls, diffs, and shell scripts. - Multi-layer sanitization (bleach, null-byte check, BIDI protection) remains active.
- Changed Claude hooks path to .claude/hooks and updated settings to matcher format. - Fixed EpisodicStore duplicate detection by ensuring stable JSON key ordering and timestamp validation. - Expanded DecisionStream/Content schemas with draft/accepted statuses for unified lifecycle. - Added scripts/repair_db.py for episodic database maintenance. - Updated documentation and integration tests.
- Added Stop hook support for instant history synchronization. - Implemented hybrid deduplication (UUID + content hash) in bridge-context. - Fixed bash hook scripts to correctly pipe stdin when PYTHONPATH is set. - Ensured string normalization for message content to prevent TypeErrors. - Added global sys/os imports in cli.py for robust hook execution.
- LLMEnricher: Added buffered CLI calling via temporary files for reliable Termux/Mobile operation. - LLMEnricher: Implemented evidence crystallization logic (save total_evidence_count and truncate ID list to last 5). - LLMEnricher: Clear raw procedural steps after successful text conversion to reduce file size. - VectorStore: Fixed llama-cpp-python detection bug (was checking uninitialized global). - Unified 'enrichment_status="pending"' across all reasoning engines. - Added scripts for rebuilding and enriching hypotheses.
- VectorStore: Added threading.Lock to GGUFEmbeddingAdapter to prevent 'Illegal instruction' crashes during concurrent calls. - Memory: Refactored record_decision to ensure active conflict checks are always enforced, regardless of vector store state. - Performance: Optimized record_decision benchmark to measure clean storage I/O (Git+SQLite) without similarity overhead. - Tests: Hardened test_conflict_injection against accidental vector similarity auto-supersedes. - Tests: Fixed LLM enrichment mock to match new file-based CLI output capture logic. - Tests: Added OperationalError handling for storage path robustness in Termux environment.
- pyproject.toml: Added filterwarnings to ignore unraisable AttributeError from llama_cpp during model destruction. - tests: Reduced seeding data in performance benchmarks from 20 to 5 records to speed up CI runs. - pytest: Filtered PytestUnraisableExceptionWarning to clean up logs and reduce teardown overhead.
- tests/core/test_bridge.py: Changed fixtures to 'module' scope to avoid redundant Memory initialization. - tests/core/audit/test_concurrency.py: Reduced iteration counts and sleep times while preserving lock verification. - tests/core/audit/test_smoke.py: Mocked LLMEnricher to test orchestration without heavy ML overhead. - tests/core/audit/test_properties.py: Reduced Hypothesis max_examples to eliminate redundant repository setups. - .github/workflows/benchmarks.yml: Updated benchmark file path to match current structure.
…owledge types - BackgroundWorker: Now calls 'run_maintenance()' instead of individual methods to ensure the enrichment queue is processed asynchronously. - Transitions: Relaxed the I1 immutability constraint to allow the enrichment worker to safely update the 'rationale' field of decisions. - LLMEnricher: Fixed the CLI prompt execution to explicitly force the 'gemini-2.5-flash-lite' model and avoid resource exhaustion. - LLMEnricher: Ordered evidence event IDs to process oldest first and limited the log payload to 100 events to prevent ARG_MAX overflow. - EpisodicStore: Fixed a Row conversion TypeError that broke the 'get_by_ids' method and prevented log fetching.
…rationale - Schemas: Added 'compressive_rationale' and 'total_evidence_count' fields for token-efficient context injection. - LLMEnricher: Implemented iterative processing in 100-event chunks with cumulative rationale building. - LLMEnricher: Added structured output parsing for 'Goal' (content), 'Rationale', and 'Compressive Summary' (exactly 3 sentences). - LLMEnricher: Enforced absolute paths in technical notes for reliable log access via 'cat'. - Transitions: Updated I1 validation to allow updates to content, rationale, and compressive_rationale during enrichment status 'pending'. - Bridge: Optimized context injection to prioritize 'compressive_rationale', significantly reducing token usage. - BackgroundWorker: Updated heartbeat to call 'run_maintenance()', enabling asynchronous enrichment processing. - EpisodicStore: Fixed Row conversion TypeError in 'get_by_ids' method. - SemanticStore: Enhanced 'update_decision' to correctly sync top-level YAML fields (title, content) with the SQLite index.
…ertion * Replaced `np.vstack` on every array insertion with a lazy memory buffer (`self._unmerged_vectors`) * Vectors are now batched into a single `np.vstack` block append before critical operations (`search`, `save`, `compact`, `_build_annoy_index`) * Massively reduces CPU overhead and avoids O(N^2) repeated array copying * Benchmark dropped from ~26 seconds to ~9.4 seconds for 10,000 array appends.
Contributor
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
…neck fix * Replaced `np.vstack` on every array insertion with a lazy memory buffer (`self._unmerged_vectors`) * Vectors are now batched into a single `np.vstack` block append before critical operations (`search`, `save`, `compact`, `_build_annoy_index`) * Massively reduces CPU overhead and avoids O(N^2) repeated array copying * Benchmark dropped from ~26 seconds to ~9.4 seconds for 10,000 array appends. * Fixed syntax error caused by Safety missing the database locally.
…neck fix * Replaced `np.vstack` on every array insertion with a lazy memory buffer (`self._unmerged_vectors`) * Vectors are now batched into a single `np.vstack` block append before critical operations (`search`, `save`, `compact`, `_build_annoy_index`) * Massively reduces CPU overhead and avoids O(N^2) repeated array copying * Benchmark dropped from ~26 seconds to ~9.4 seconds for 10,000 array appends. * Fixed CI failures: - Fixed syntax error caused by Safety missing the database locally. - Added permissions and bounds checking to GitHub Script in `.github/workflows/minimal-ci.yml` so it functions correctly for PR actions that need write access or push events without issue numbers.
🔒 Security Scan Results\n\n### Bandit (SAST)\n- 0 HIGH severity issues\n\n### Safety (Dependencies)\n- 0 known vulnerabilities\n\n### Pip-audit (Dependencies)\n- 0 vulnerable packages\n |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 What: Deferred
np.vstackarray copies inVectorStore.add_documentsby buffering vectors into a standard python list (self._unmerged_vectors), performing a single batched array copy lazily beforesaveorsearchoperations.🎯 Why: Repeatedly copying the entire vector index (
np.vstack) for every single sequential insert (e.g.memory.record_decision()) caused an O(N²) memory/CPU bottleneck, dragging down batch ingestion and background sync tasks.📊 Impact: Massively speeds up batch document ingestion (e.g.,
sync_gitor large memory restorations).🔬 Measurement: A benchmarking script inserting 10,000 vectors one by one natively in Python dropped from ~26s down to ~9.4s.
Also added
bolt.mddocumentation explaining the risk ofnp.vstackinside single item loops and the importance of lazy batching.PR created automatically by Jules for task 1455594555211211091 started by @sl4m3