Next 2026-05-15

.ai/skills/cluster-ssh/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: cluster-ssh
+description: SSH into a cardano-parts cluster's hosts over shared ControlMaster sockets to run read-only on-host diagnostics (journalctl, systemctl, zfs, cardano db tools, etc.), and move bulk state directly between hosts with wush when needed. Use when investigating node or host behavior directly on the machine — when telemetry (see the monitoring-query skill) isn't enough, or you need to inspect units, files, or local state. Requires an SRE to have opened the masters first.
+---
+
+# On-host access to cardano-parts cluster machines
+
+This skill ships with the cardano-parts template and is deployment-agnostic: the mechanism (the
+`ai-ssh` recipes, ControlMaster sockets, `nss_wrapper`) is identical across cardano-parts clusters.
+Only the host names differ — discover them per cluster (see §3). Example hostnames below are
+illustrative.
+
+Cluster hosts are reached through **AWS SSM** (`ProxyCommand … aws ssm start-session …`), so an
+agent sandbox — which usually has no AWS creds and no outbound AWS network — **cannot dial hosts
+itself**. Instead, an SRE opens persistent SSH **ControlMaster** sessions, and the agent rides the
+shared sockets. The agent never needs AWS creds: the master process holds the SSM tunnel.
+
+This is the escalation path from the `monitoring-query` skill: use telemetry first; come here when
+you need to read units/files/DB state on the box. **Default to read-only commands** (`journalctl`,
+`systemctl status`, `zfs list`, `jq`, `cat`, `ls`, `db-analyser --help`). Anything that mutates
+state (restart, edit, destroy, write) must be confirmed with the SRE first.
+
+## 1. SRE opens the masters (their side)
+
+From the repo root the SRE runs the `ai-ssh` recipe with the host(s) to expose:
+
+```sh
+just ai-ssh <host> [<host> ...]      # e.g. just ai-ssh group1-role-a-1
+just ai-ssh-list                     # show which masters are live
+just ai-ssh-close [<host> ...]       # close all, or the named hosts
+```
+
+`ai-ssh` (re)generates an agent ssh config and opens a backgrounded master per host. It prints the
+two paths the agent needs:
+
+- **Agent config:** `$AI_SSH_DIR/ssh_config_ai`
+- **Socket dir:** `$AI_SSH_DIR` (sockets named `cm-<hostname>`)
+
+`AI_SSH_DIR` defaults to a repo-local `.ssh-ai/` but the SRE may set it to a path your sandbox can
+read (e.g. a mounted share). **Don't assume a path — use the one the recipe printed**, or ask the
+SRE for `AI_SSH_DIR`. The config embeds an absolute `ControlPath`, so it resolves to the same socket
+from your sandbox despite a different `$HOME`/cwd.
+
+## 2. Agent rides the masters (your side)
+
+Connect with the generated config; the master does the I/O, so no creds/tunnel are needed:
+
+```sh
+CFG="$AI_SSH_DIR/ssh_config_ai"          # the path the recipe printed
+ssh -F "$CFG" <host> '<read-only command>'
+```
+
+**Always verify the master is up first** (non-dialing — fails fast if the SRE hasn't opened it):
+
+```sh
+ssh -F "$CFG" -O check <host>            # "Master running (pid=…)" = good
+```
+
+### If your sandbox is unprivileged with no passwd entry
+
+OpenSSH aborts with `No user exists for uid …` when the sandbox uid isn't in `/etc/passwd`. Fix with
+`nss_wrapper` (a fake passwd/group + `LD_PRELOAD`). One-time bootstrap, then wrap every ssh call:
+
+```sh
+NW=$(nix build --no-link --print-out-paths nixpkgs#nss_wrapper)
+printf 'agent:x:%s:%s::%s:/bin/sh\n' "$(id -u)" "$(id -g)" "$HOME" > /tmp/nwpasswd
+printf 'agent:x:%s:\n' "$(id -g)" > /tmp/nwgroup
+
+# /tmp/sshw — invoke as:  bash /tmp/sshw <host> '<cmd>'
+cat > /tmp/sshw <<EOF
+#!/usr/bin/env bash
+exec env LD_PRELOAD="$NW/lib/libnss_wrapper.so" \\
+  NSS_WRAPPER_PASSWD=/tmp/nwpasswd NSS_WRAPPER_GROUP=/tmp/nwgroup \\
+  USER=agent LOGNAME=agent ssh -F "$CFG" "\$@"
+EOF
+bash /tmp/sshw <host> -O check          # verify, then run real commands
+```
+
+If your sandbox already has a passwd entry, skip the wrapper and use `ssh -F "$CFG"` directly.
+
+## 3. Finding hosts
+
+```sh
+just ssh-list name '.*'                  # all node names
+just ssh-list name 'preprod.*'           # filtered
+nix eval --json '.#nixosConfigurations' --apply builtins.attrNames | jq
+```
+
+Naming: `<env><n>-<role>-<az>-1`, e.g. `leios1-rel-a-1`, `preprod2-bp-b-1`. Block producers
+(`-bp-`) sit behind their relays (`-rel-`); an SRE may only have opened a subset — check
+`just ai-ssh-list`.
+
+## Moving bulk data between hosts (`wush`)
+
+The agent reaches hosts only through the SRE's SSM/ControlMaster path, and cluster hosts generally
+**cannot SSH each other directly** (SSM-only; inter-host port 22 is typically closed). So relaying a
+large file host→agent→host is slow and fragile — it crosses the SSM tunnel twice and dies if a master
+flaps. When you need to move bulk state between two machines — e.g. transplanting a node's database /
+state directory from a healthy donor onto a broken one — use **`wush`** if it's installed
+(`command -v wush`; on NixOS it can be added to system packages or fetched via `nix`).
+
+`wush` is a WireGuard-powered peer-to-peer transfer + shell tool. It stands up its own encrypted
+overlay between the two hosts (direct UDP, or DERP-relayed when there's no direct path), so bytes flow
+**host-to-host**, bypassing both the SSM bottleneck and the missing inter-host SSH — typically orders
+of magnitude faster than relaying through the agent.
+
+**Model:** one side runs `wush serve` (it prints an `--auth-key`); the other runs a client
+(`wush rsync`, `wush cp`, `wush ssh`, `wush port-forward`) with that key. A leading `:` denotes a path
+on the *server*:
+
+```sh
+# On the SERVER host (the one you transfer to/from). Run it backgrounded; it prints the auth key
+# on the line just after "Picked DERP region …".
+wush serve                                                    # → auth key: <base58 string>
+
+# On the CLIENT host, with that key (extra rsync flags go after `--`):
+wush rsync --auth-key <key> :/srv/path  /local/path  -- -a    # pull  (server → client)
+wush rsync --auth-key <key> /local/path :/srv/path   -- -a    # push  (client → server)
+```
+
+Driving it from the agent: start `wush serve` on one host as a backgrounded SSM session, read its auth
+key from the output, then run the `wush rsync` client on the other host. One running serve + key
+handles multiple parallel clients, so a single donor can fan its data out to several recipients at once.
+
+**Transplanting live state safely:**
+- Briefly **stop the service on the source** for an on-disk-consistent copy, then restart — never copy
+  a database that's being written. To shorten the source's downtime, stage a scratch copy while it's
+  stopped and serve the transfer from that. A plain `cp` of the state dir *may* be near-instant on ZFS
+  (a `copy_file_range` block-clone) — but only with OpenZFS ≥ 2.2, the `block_cloning` pool feature,
+  `zfs_bclone_enabled=1`, and a `cp` that does reflinks (GNU coreutils ≥ 9); otherwise it's a full
+  copy. Don't assume it — check (`zpool get feature@block_cloning`, `cat
+  /sys/module/zfs/parameters/zfs_bclone_enabled`), or just keep the source stopped for the whole
+  transfer. For a guaranteed-instant *and* consistent point-in-time source, `zfs snapshot` works
+  regardless of `cp` behavior — but remember to `zfs destroy` it afterward.
+- `rsync -a` preserves owner/perms; cluster hosts share the same service uid/gid, so ownership comes
+  across correctly. State dirs carry no per-host identity (keys live elsewhere), so they're safe to
+  clone between same-network nodes.
+- This **mutates the target** — confirm with the SRE first; read-only-by-default still applies.
+- **Cleanup:** stop the server with **`pkill -x wush`** — *not* `pkill -f "wush serve"`, which also
+  matches your own command line and kills your shell. Remove any scratch copies you staged.
+
+## Norms & gotchas
+
+- **Read-only by default.** Surface anything mutating to the SRE before running it.
+- **Hosts are UTC** — no `--utc` needed for `journalctl -S/-U`.
+- **journald JSON lines are duplicated (×2)** in some setups — sanity-check counts.
+- **Capture long output to a file, then grep/parse it** — don't re-run expensive `journalctl`/`curl`.
+- Node log format (JSON vs human) is a per-host tracer choice and can change on redeploy — sample a
+  line before parsing (same caveat as the `monitoring-query` skill).
+- Masters expire after `AI_SSH_PERSIST` (default 2h); if `-O check` fails, ask the SRE to re-run
+  `just ai-ssh <host>`.
+- **Don't relay bulk data through your sandbox** — to move large state between hosts use `wush`
+  (see *Moving bulk data between hosts*), which goes host-to-host and survives master flaps.

.ai/skills/leios-repair/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: leios-repair
+description: Diagnose and repair Leios cluster forks and stuck block producers — detect whether the network is forked (which nodes, what trigger), and reseed a forked/behind node's chain state from a healthy donor relay so it rejoins the main chain. Use when a Leios environment shows a fork, a BP stuck on its own tip, a node falling behind, or an endorser-block (EB) production stall. Requires on-host access via the cluster-ssh skill (SRE opens masters first).
+---
+
+# Repairing Leios forks & stuck nodes
+
+Leios (the leios* environments) hits a recurring class of **non-self-healing forks**: a block
+producer ends up on its own branch and cannot re-converge to the main chain on its own. Until the
+upstream fix lands (ouroboros-leios issue #890 / PR #2073), **operator reseed is the remedy each
+time**. This skill is the diagnose→fix playbook.
+
+- **Access** is via the **`cluster-ssh`** skill (ControlMaster sockets + `/tmp/sshw`, `wush` for
+  host-to-host bulk transfer). Read it first; this skill assumes `bash /tmp/sshw <host> '<cmd>'` works.
+- **Telemetry** is via the **`monitoring-query`** skill (Loki/Mimir) — use it for the fleet-wide
+  first look (block heights, EB counters, density) before going on-host.
+- **Read-only by default.** The reseed steps here **mutate** a node — confirm with the SRE first
+  (see *Mutation policy*).
+
+> **Status & maintenance** (this skill tracks a moving target — keep it current):
+> - *Last verified:* 2026-07-03, against the current Leios playground deploy.
+> - *Reseed still required:* YES — forks are non-self-healing pre ouroboros-leios #890 / PR #2073.
+>   When that fix + pin bump lands, re-test whether forks self-heal and **downgrade/retire the reseed
+>   loop** here accordingly.
+> - *Known-open behaviors:* the fork families in §4; EB-production stall (§1c). Add new triggers to §4
+>   as they're found (name the trigger + a triage grep) and link the incident write-up.
+> - *When the network upgrades or storage/paths/units change:* re-confirm §2 (storage layout), the
+>   magic/socket/cli constants, and that `recover_bp.sh`'s assumptions still hold — update both files
+>   together. Bump *Last verified* whenever you re-check against a new deploy.
+
+Deployment specifics below (socket path, network magic, storage paths) are for the current Leios
+playground deploy — **confirm them** if the deploy changed (`systemctl cat cardano-node`, `ls`).
+Network magic = **164**; node socket = `/run/cardano-node/node.socket`; cli =
+`/run/current-system/sw/bin/cardano-cli`. Hosts: `leios<N>-{bp,rel}-<az>-1` (+ dbsync/faucet/centrifuge
+on leios1). Relays track the main chain; BPs (`-bp-`) forge.
+
+## 1. Is it actually forked? (diagnose before touching anything)
+
+A BP briefly **ahead** of the relays (it just forged) is normal and self-resolves. A **fork** is when
+a node sits on a hash the main chain never adopts. Distinguish them:
+
+**a) Tip vs relays, sampled over ~40s.** Healthy = BPs converge to the relay hash within a slot or
+two. Fork = a BP's hash never appears on the main chain while relays advance on *other* hashes.
+```sh
+for h in leios1-rel-a-1 leios1-bp-a-1 leios2-bp-b-1 leios3-bp-c-1; do
+  printf "%-14s " "$h"
+  bash /tmp/sshw "$h" 'export CARDANO_NODE_SOCKET_PATH=/run/cardano-node/node.socket; \
+    cardano-cli query tip --testnet-magic 164 2>/dev/null | grep -oE "\"block\": [0-9]+|[0-9a-f]{16}" | tr "\n" " "'; echo
+done
+# repeat a few times (sleep 8) — see whether BP hashes ever match the main chain
+```
+
+**b) Forger identity on each BP's own chain** (the decisive test). On a forked BP, recent
+`AddedToCurrentChain` blocks are forged **only by itself** (one `issuerHash`); a healthy node adopts
+**many foreign** forgers. This is a multi-operator net (~11 pools) — our 3 BPs are a subset.
+```sh
+bash /tmp/sshw leios3-bp-c-1 'journalctl -u cardano-node --no-pager -S "8 min ago" -o cat 2>/dev/null \
+  | grep -a "\"kind\":\"AddedToCurrentChain\"" | grep -aoE "\"issuerHash\":\"[0-9a-f]{10}" | sort | uniq -c'
+# only-own issuerHash + tip never matching main chain = SOLO FORK → reseed that BP.
+```
+
+**c) EB-production stall** (a *different* failure — NOT a fork). A BP can be on the main chain for RBs
+yet stop forging **endorser blocks**. Compare the counter across BPs (Mimir / monitoring-query):
+`cardano_node_metrics_Forge_endorser_block_total_count_counter{environment="leios",instance=~".*-bp-.*"}`
+— a flat line while peers climb (~3 EB/hr/pool) = EB stall. **Do NOT reseed for this** (the RB chain
+is fine); investigate the EB-forge path / restart as a last resort.
+
+**d) Wedged *behind* (BP _or_ relay).** A node — including a **relay** — can stall *behind* the main
+chain instead of forking ahead: frozen tip well below the relays, `LeiosBlockPointMissing` present,
+and **0** `AddedToCurrentChain` / `CompletedBlockFetch` / `DownloadedHeader` (it isn't even pulling
+headers). That's the EB-closure wedge (§4), not a competing branch — the fix is still a reseed. A
+wedged *relay* may leave its group's BP healthy (the BP sources blocks from other relays), but it
+does disqualify that relay as a reseed donor (§3).
+
+## 2. Storage layout (what gets reseeded)
+
+Leios nodes use split storage:
+- `/var/lib/cardano-node/db-leios` — RB/ledger chain DB (EBS).
+- `/ephemeral/cardano-node/leios.db` (+ `-wal`/`-shm`) — Leios EB/vote DB (ephemeral disk).
+
+Donor relays publish a combined snapshot at `/ephemeral/nginx-artifacts/leios.full.tar.zst`
+(+ `.sha256`, `.meta.json`) — top-level `db-leios/` + `leios.db`. **Only `-rel-*-1` relays carry the
+artifact** (check per group); it's an **hourly** ZFS snapshot.
+
+## 3. Reseed a forked/behind node (`recover_bp.sh`)
+
+`recover_bp.sh` (in this skill dir) does: stop node → `wush` the donor's artifact host-to-host →
+verify sha → swap DBs (old → `*.fork-bak`, non-destructive) → restart. Works for BPs **and** dbsync
+hosts (db-sync auto-reconnects and rolls forward; no Postgres reseed needed).
+
+**Pick a donor** = *any* relay **on the main chain** with a **fresh** artifact — normally the
+target's own group relay, but **cross-group is fine** (the artifact is main-chain state, a coherent
+same-relay `db-leios`+`leios.db` pair). **Never use a wedged relay as donor:** a relay stalled behind
+(§1d) keeps publishing, so its artifact is stale/behind — reseed *it* from a healthy relay in another
+group instead (e.g. leios3-rel-c-1 → leios2-rel-b-1, done 2026-07-03):
+```sh
+for r in leios1-rel-a-1 leios2-rel-b-1 leios3-rel-c-1; do printf "%-16s " "$r"; \
+  bash /tmp/sshw "$r" 'ls --time-style=+%H:%M:%SZ -l /ephemeral/nginx-artifacts/leios.full.tar.zst 2>/dev/null | awk "{print \$6}"; \
+    grep -oE "\"snapshot_created\":[^,]+" /ephemeral/nginx-artifacts/leios.full.tar.zst.meta.json 2>/dev/null'; echo; done
+```
+
+**Run it** (confirm with SRE first — this mutates the target):
+```sh
+bash .ai/skills/leios-repair/recover_bp.sh leios3-bp-c-1 leios3-rel-c-1   # <target> <donor-relay>
+```
+(The agent typically runs a copy from `/tmp`; keep it there or reference the skill path.)
+
+**Reseed only the FORKED nodes; leave healthy ones alone.** If several BPs are forked, reseed each
+from its group relay (sequential is safest; the script `pkill`s `wush` on the donor between runs).
+
+**Verify convergence** — wait for the main chain to advance ~8 blocks and confirm every reseeded node
+matches the relay tip hash, and is adopting **foreign** blocks again (§1b). Only then call it fixed.
+
+### Whack-a-mole caveat
+Taking BPs offline for reseed disturbs the small cluster's block propagation, which can tip the
+**next** BP into the same fork (observed 2026-07-01: reseeding two BPs → the third forked). **Always
+re-check ALL BPs after a reseed round**, not just the ones you reseeded.
+
+## 4. Known fork/stall families (for triage & the consensus team)
+
+All share the end-state (solo fork, non-self-healing, reseed required) but differ by trigger — name
+the trigger in the incident:
+- **EB-closure / data-availability wedge** — `LeiosBlockPointMissing` in the journal; RB selection
+  blocked on a missing EB body (`StoreButDontChange` *caused by* the missing point). Triggers seen:
+  size-0 EB offer, EB slot-identity mismatch, tx-closure never requested. **Hits relays too**, not
+  just BPs (2026-07-03: leios2-rel-b-1 wedged 20 behind, 0 header/fetch/adopt) — a relay wedges
+  *behind* rather than forking ahead (§1d).
+- **RB chain-selection non-convergence** (2026-07-01) — **zero** `LeiosBlockPointMissing`,
+  `weightBoost`=0 everywhere; competing same-height blocks + each BP forges ahead on its own branch +
+  self-favoring VRF tiebreak → `StoreButDontChange` on the main-chain blocks and never re-converges.
+  `CompletedBlockFetch` present (it *does* fetch the main blocks; it just won't switch).
+- **EB-production stall** — RBs fine, `Forge_endorser_block_total_count_counter` flat (§1c). Not a
+  fork; reseed won't help.
+
+Quick triage greps in the fork window (hosts are UTC):
+```sh
+bash /tmp/sshw <bp> 'journalctl -u cardano-node --no-pager -S "<start>" -U "<end>" -o cat 2>/dev/null \
+  | grep -acE "LeiosBlockPointMissing"'   # >0 => closure wedge; 0 => chain-select family
+```
+
+## Mutation policy
+Diagnosis (§1) is read-only — run freely. The reseed (§3) **stops, swaps DB, and restarts** a node:
+**confirm with the SRE before running**, consistent with `cluster-ssh`. It is non-destructive (old
+DB kept at `*.fork-bak`); once convergence is verified and stable, the `*.fork-bak` dirs can be
+removed to reclaim space.
+
+## Gotchas
+- `density_real` is a **trailing average** — it lags a fixed fork/load window; check live per-BP
+  forge counts before concluding "nothing is forging".
+- SSM masters expire (`AI_SSH_PERSIST`, default 2h); `-O check` first, ask SRE to re-open if down.
+- Forks do **not** self-heal pre-#2073 — expect recurrence; a reseed is a stopgap, not a fix.
+- Don't reseed a node that's merely *ahead* of the relays (it just forged) — confirm a true fork (§1).