Merge branch 'feat/local-search-index'

Author: Tommy

CHANGELOG.md

@@ -5,6 +5,70 @@ All notable changes to Earth Reclaim are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] - 2026-06-30
+
+### Added
+
+- **Local search index — "Google, but completely local."** Type a query and get
+  fast results that are scraped, indexed, and grep-able on your device. It fuses the
+  local SearXNG meta-search, the web scraper, and the AI curator behind one
+  `local_search` command plus a new unified **FTS5 + vector** index with a fusion
+  ranker. The first search for a topic is slow (live discover → scrape → index);
+  every search after is instant off a warm local index — and the index only ever
+  contains things you actually searched (strong privacy property).
+  - **Two-speed streaming:** SearXNG-speed snippets paint immediately, then scraped
+    + indexed results stream in, then the list re-orders to a fused ranking.
+  - **Hybrid ranking:** FTS5/BM25 ⊕ vector cosine ⊕ SearXNG position ⊕ a private
+    click-log, fused with Reciprocal Rank Fusion.
+  - **Lifecycle ladder & retention tiers:** browse → auto-cache (TTL'd, no curation
+    cost) → favorite/pin (permanent, curated) → archived (summary kept, body/FTS/
+    embeddings dropped) → forgotten. Auto-GC only ever touches ephemeral/cache;
+    pinned/archived are protected. Login/credential pages are never cached or indexed.
+  - **Favorites = pins, one source of truth:** a pin control (distinct from the
+    bookmark star) in the address bar, History rows, and search results, all reading
+    the same retention tier. Bookmarks stay separate (URL-only, no indexing).
+  - **Review-pinned panel:** the curator *proposes* prune candidates (disuse, age,
+    semantic redundancy; a dead upstream is protected, not pruned) and *you dispose* —
+    nothing pinned is ever silently removed; the default destructive action is archive.
+- **Crawler results in unified search.** Pages from the web scraper's crawl jobs now
+  appear in search results, fused via the same RRF — read-only, capped per domain,
+  with a "from crawl: \<job\>" badge. Crawler storage is never modified.
+- **Comments & discussions search.** A typed-content filter (All / Comments &
+  discussions / Comments only) with per-platform adapters that pull posts and comments
+  from Reddit and forums (Discourse / Stack Exchange / generic) by default, plus
+  YouTube and TikTok via yt-dlp. Instagram/Facebook are best-effort and **off by
+  default** (public, logged-out only). An optional **opt-in** "use my own session"
+  toggle (default off, with a blunt Terms-of-Service warning) exists for the social
+  adapters — no credential automation, ever.
+- **Pagination** — "More results / next page" fetches a genuinely different page of
+  results (SearXNG `pageno`).
+- **Per-profile Local-AI settings** — the knowledge curator and assistant toggles now
+  persist per profile in the database.
+- README screenshots (hero + gallery).
+
+### Changed
+
+- **Search page UX:** the search controls (retention, kinds, sources, ranking-signals
+  debug, DuckDuckGo) are now an always-visible bar — selectable *before* a search. The
+  domain manager collapses while searching to keep results in focus, and the address
+  bar placeholder is now "Search or enter a URL to visit".
+- **Feature naming:** EarthMemory → **Journal**, EarthMultiMedia → **Media**, and the
+  local AI is now **Sage** (summarizes what you read into your Journal and answers from
+  it & the web, fully offline). The About panel is now a right-side drawer.
+
+### Fixed
+
+- **Secondary windows rendered as a blank "The URL can't be shown" box in packaged
+  builds.** Programmatic windows now match the main window's incognito web context, so
+  the app's asset protocol resolves (regression from the v1.0.8 single-instance change).
+- **The knowledge curator switched itself back on after every restart.** Its on/off
+  state was kept in the incognito WebView's localStorage (wiped each launch); it's now
+  persisted per profile in the database.
+- **Crawled pages polluted unrelated searches** (e.g. "best grapheneos phone" surfaced a
+  crawled site with none of those words). The crawler fan-in now requires *all* query
+  terms, is capped per domain, and live results are kept after ranking completes.
+- A GC scheduler panic on startup (background task spawned outside the Tokio runtime).
+
 ## [1.0.8] - 2026-06-29
 
 ### Security

README.md

@@ -6,6 +6,17 @@
   [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
   [![Rust](https://img.shields.io/badge/rust-%23000000.svg?style=flat&logo=rust&logoColor=white)](https://www.rust-lang.org/)
   [![Tauri](https://img.shields.io/badge/tauri-%2324C8DB.svg?style=flat&logo=tauri&logoColor=%23FFFFFF)](https://tauri.app/)
+
+  <br>
+
+  <img src="docs/Main.png" alt="Reclaim — privacy-first desktop browser" width="90%">
+
+  <table>
+    <tr>
+      <td width="50%"><img src="docs/Search.png" alt="Local search index"></td>
+      <td width="50%"><img src="docs/Nobara.png" alt="Reclaim running on Linux"></td>
+    </tr>
+  </table>
 </div>
 
 ---
@@ -27,9 +38,17 @@ Everything below runs **on your machine**. The only outbound traffic is the page
 - **Password‑protected bookmarks** — bookmarks are encrypted at rest, with an optional Argon2id password gate for a private set.
 - **NoScript** — JavaScript is blocked by default and trusted **per site** (persistent or session‑only); a web‑process extension blocks untrusted third‑party requests.
 - **Privacy protection** — third‑party cookie blocking, tracking prevention (ITP), user‑agent spoofing, and incognito (ephemeral, nothing on disk).
-- **Your own search engine** — build a private index from the sites *you* curate and **scrape**, plus community trust/bias ratings on domains (EarthSearch). Search your own corpus, not an ad network.
+- **Your own search engine** — build a private index from the sites *you* curate and **scrape**, plus community trust/bias ratings on domains. Search your own corpus, not an ad network.
+
+### 🔍 Local Search — "Google, but completely local"
+- **Query‑driven local index** — type a query and get fast results that are **scraped, indexed, and grep‑able on your device**. It fuses local SearXNG meta‑search, the web scraper, and the AI curator behind one index (**FTS5 full‑text ⊕ vector embeddings**) with a fusion ranker. The index only ever contains things *you actually searched* — strong privacy by construction.
+- **Two‑speed results** — SearXNG‑speed snippets paint instantly, then scraped + indexed results stream in, then the list re‑orders to a **fused ranking** (BM25 ⊕ vector cosine ⊕ SearXNG position ⊕ a private click‑log, via Reciprocal Rank Fusion). The first search for a topic is slow; every search after is instant.
+- **Retention ladder** — browse → auto‑cache (TTL'd, no curation cost) → **favorite/pin** (permanent, curated) → archived (summary kept, body dropped) → forgotten. Auto‑GC only ever touches the cheap tiers; pins are protected. Login/credential pages are never cached or indexed.
+- **Favorites = pins, one source of truth** — a pin control (distinct from the bookmark star) in the address bar, History, and results, all reading the same tier. A **review‑pinned** panel where the curator *proposes* prune candidates (disuse, age, semantic redundancy) and *you* dispose — pins are never silently removed.
+- **Comments & discussions** — filter to comments/discussions and pull posts + comments from **Reddit** and **forums** (Discourse / Stack Exchange / generic) by default, plus **YouTube** and **TikTok** via yt‑dlp. (Instagram/Facebook are best‑effort and **off by default** — public, logged‑out only; an optional, default‑off "use my own session" toggle carries a blunt Terms‑of‑Service warning. No credential automation.)
+- **Crawler fan‑in** — pages from your Web Scraper crawl jobs join the same results (read‑only, capped per domain, badged "from crawl").
 
-### 🎬 Media (EarthMultiMedia)
+### 🎬 Media
 - **Multi‑pane viewing** — single, split (horizontal/vertical), or quad; each pane is an independent native player.
 - **Workspace tabs** — browser‑style media tabs, each with its own queue, panes, and layout.
 - **Queue** — drag‑to‑reorder, **shuffle**, repeat, and consecutive autoplay (next item plays in the pane that just finished).
@@ -43,9 +62,9 @@ Everything below runs **on your machine**. The only outbound traffic is the page
 - **Live status** — jobs run on creation (or via ▶ Run), streaming `pending → running → completed` with a live page count.
 - **Opt‑in “Add to AI memory”** — per job, also summarize each scraped page into your knowledge graph so the assistant can use it.
 
-### 🧠 Local AI (Ollama)
-- **Knowledge Curator** — summarizes **what you actually read** (an in‑page viewed‑content bridge captures only text you scrolled into view — comments included only if you reach them), not a blind re‑fetch. Skips homepages/feeds, de‑duplicates, writes a rich summary + a small excerpt. Transparent, unbiased, skips incognito.
-- **AI Assistant** — a private, streaming, multi‑session chat (ChatGPT‑style) grounded in your saved pages, media notes, and past conversations. Model auto‑selected by your GPU tier, or pick any installed model.
+### 🧠 Sage — Local AI (Ollama)
+- **Knowledge Curator** — summarizes **what you actually read** (an in‑page viewed‑content bridge captures only text you scrolled into view — comments included only if you reach them), not a blind re‑fetch, into your **Journal** (knowledge graph). Skips homepages/feeds, de‑duplicates, writes a rich summary + a small excerpt. Transparent, unbiased, skips incognito. Its on/off state is remembered **per profile**.
+- **AI Assistant** — a private, streaming, multi‑session chat (ChatGPT‑style) grounded in your Journal, media notes, and past conversations. Model auto‑selected by your GPU tier, or pick any installed model.
 - **🔬 Web Research mode** — agentic **search + read**: the assistant uses local tool‑calling to `web_search` and `fetch_url`, shows each step (🔎 searching / 📄 reading), then streams a grounded answer that **cites the URLs it actually read**. Searches route through a local **SearXNG** instance when available (fully private), or **DuckDuckGo** as a fallback. All reasoning stays on‑device.
 - **Password‑protected tab** — lock the whole Local AI / History tab behind its own unique Argon2id password (separate from the vault, media, and bookmark passwords).
 

apps/reclaim/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reclaim",
-  "version": "1.0.8",
+  "version": "1.1.0",
   "private": true,
   "type": "module",
   "scripts": {

apps/reclaim/src-tauri/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "reclaim"
-version = "1.0.8"
+version = "1.1.0"
 description = "Earth Reclaim - Privacy-first browser with local AI"
 authors = ["EarthServers"]
 license = "MIT"