feat(db): configurable Postgres statement_timeout on pool connections (#1200)

* feat(db): apply configurable Postgres statement_timeout on pool connections

Adds HINDSIGHT_API_DB_STATEMENT_TIMEOUT (default 600s, set 0 to disable).
Applied via the asyncpg pool init hook, so it only affects runtime
queries — Alembic migrations run on a separate psycopg2 engine and are
untouched.

Also fixes the ANN chunk path in the retain orchestrator to restore the
pool's configured statement_timeout rather than RESET, which would fall
back to the server default and silently drop the safety net on that
pooled connection.

* refactor(ann): drop fixed per-query timeout on compute_semantic_links_ann

Now that the asyncpg pool applies a Postgres statement_timeout to every
connection (HINDSIGHT_API_DB_STATEMENT_TIMEOUT, default 600s), the ANN
path can be treated like any other query — no need for the 300s asyncpg
per-query timeout or the orchestrator's SET/restore dance around the
pool's default.

* chore: regenerate hindsight-docs skill to pick up configuration doc change

Re-runs scripts/generate-docs-skill.sh so the skill reference mirror
matches the HINDSIGHT_API_DB_STATEMENT_TIMEOUT row added in
hindsight-docs/docs/developer/configuration.md.

Also picks up unrelated drift (openapi.json version bump, changelog
index) that had accumulated on main.

Author: nicoloboschi

hindsight-api-slim/hindsight_api/config.py

@@ -375,6 +375,7 @@ def normalize_config_dict(config: dict[str, Any]) -> dict[str, Any]:
 ENV_DB_POOL_MAX_SIZE = "HINDSIGHT_API_DB_POOL_MAX_SIZE"
 ENV_DB_COMMAND_TIMEOUT = "HINDSIGHT_API_DB_COMMAND_TIMEOUT"
 ENV_DB_ACQUIRE_TIMEOUT = "HINDSIGHT_API_DB_ACQUIRE_TIMEOUT"
+ENV_DB_STATEMENT_TIMEOUT = "HINDSIGHT_API_DB_STATEMENT_TIMEOUT"
 
 # Worker configuration (distributed task processing)
 ENV_WORKER_ENABLED = "HINDSIGHT_API_WORKER_ENABLED"
@@ -607,6 +608,7 @@ def normalize_config_dict(config: dict[str, Any]) -> dict[str, Any]:
 DEFAULT_DB_POOL_MAX_SIZE = 100
 DEFAULT_DB_COMMAND_TIMEOUT = 60  # seconds
 DEFAULT_DB_ACQUIRE_TIMEOUT = 30  # seconds
+DEFAULT_DB_STATEMENT_TIMEOUT = 600  # seconds (Postgres statement_timeout applied on every pool connection; 0 disables)
 
 # Worker configuration (distributed task processing)
 DEFAULT_WORKER_ENABLED = True  # API runs worker by default (standalone mode)
@@ -1064,6 +1066,7 @@ class HindsightConfig:
     db_pool_max_size: int
     db_command_timeout: int
     db_acquire_timeout: int
+    db_statement_timeout: int
 
     # Worker configuration (distributed task processing)
     worker_enabled: bool
@@ -1671,6 +1674,7 @@ def from_env(cls) -> "HindsightConfig":
             db_pool_max_size=int(os.getenv(ENV_DB_POOL_MAX_SIZE, str(DEFAULT_DB_POOL_MAX_SIZE))),
             db_command_timeout=int(os.getenv(ENV_DB_COMMAND_TIMEOUT, str(DEFAULT_DB_COMMAND_TIMEOUT))),
             db_acquire_timeout=int(os.getenv(ENV_DB_ACQUIRE_TIMEOUT, str(DEFAULT_DB_ACQUIRE_TIMEOUT))),
+            db_statement_timeout=int(os.getenv(ENV_DB_STATEMENT_TIMEOUT, str(DEFAULT_DB_STATEMENT_TIMEOUT))),
             # Worker configuration
             worker_enabled=os.getenv(ENV_WORKER_ENABLED, str(DEFAULT_WORKER_ENABLED)).lower() == "true",
             worker_id=os.getenv(ENV_WORKER_ID) or DEFAULT_WORKER_ID,

hindsight-api-slim/hindsight_api/engine/memory_engine.py

@@ -482,6 +482,7 @@ def __init__(
         self._pool_max_size = pool_max_size if pool_max_size is not None else config.db_pool_max_size
         self._db_command_timeout = db_command_timeout if db_command_timeout is not None else config.db_command_timeout
         self._db_acquire_timeout = db_acquire_timeout if db_acquire_timeout is not None else config.db_acquire_timeout
+        self._db_statement_timeout = config.db_statement_timeout
         self._run_migrations = run_migrations
         self._retain_entity_lookup = config.retain_entity_lookup
 
@@ -1848,6 +1849,8 @@ async def verify_llm():
         # Create connection pool
         # For read-heavy workloads with many parallel think/search operations,
         # we need a larger pool. Read operations don't need strong isolation.
+        stmt_timeout_s = self._db_statement_timeout
+
         async def _init_connection(conn: asyncpg.Connection) -> None:
             # SET (not SET LOCAL) so it persists for the connection lifetime.
             # ef_search=200 improves HNSW recall quality for the per-fact_type
@@ -1857,6 +1860,12 @@ async def _init_connection(conn: asyncpg.Connection) -> None:
             except Exception:
                 logger.debug("Could not set hnsw.ef_search — extension may not support it")
 
+            # Server-side safety net for runaway queries. Migrations use a
+            # separate SQLAlchemy/psycopg2 engine, so long-running DDL is
+            # unaffected. 0 disables.
+            if stmt_timeout_s > 0:
+                await conn.execute(f"SET statement_timeout = '{stmt_timeout_s}s'")
+
         self._pool = await asyncpg.create_pool(
             self.db_url,
             min_size=self._pool_min_size,

hindsight-api-slim/hindsight_api/engine/retain/link_utils.py

@@ -812,7 +812,6 @@ async def compute_semantic_links_ann(
                 bank_id,
                 fact_type,
                 top_k,
-                timeout=300,  # ANN on large banks can take minutes
             )
             logger.debug(f"[ANN] fact_type={fact_type}: {len(ft_rows)} rows in {time_mod.time() - t_query:.3f}s")
             rows.extend(ft_rows)

hindsight-api-slim/hindsight_api/engine/retain/orchestrator.py

@@ -753,7 +753,6 @@ async def _process_ann_chunk(chunk_idx: int) -> None:
         async with ann_semaphore:
             t0 = time.time()
             async with acquire_with_retry(pool) as conn:
-                await conn.execute("SET statement_timeout = '300s'")
                 ann_links = await compute_semantic_links_ann(
                     conn,
                     bank_id,
@@ -766,7 +765,6 @@ async def _process_ann_chunk(chunk_idx: int) -> None:
                 if ann_links:
                     await _bulk_insert_links(conn, ann_links, bank_id=bank_id)
                 chunk_link_counts[chunk_idx] = len(ann_links)
-                await conn.execute("RESET statement_timeout")
             logger.info(
                 f"[streaming] Final ANN chunk {chunk_idx + 1}/{num_chunks}: "
                 f"{len(ann_links)} links in {time.time() - t0:.3f}s"

hindsight-docs/docs/developer/configuration.md

@@ -45,8 +45,9 @@ Migrations will automatically create the schema if it doesn't exist and create a
 |----------|-------------|---------|
 | `HINDSIGHT_API_DB_POOL_MIN_SIZE` | Minimum connections in the pool | `5` |
 | `HINDSIGHT_API_DB_POOL_MAX_SIZE` | Maximum connections in the pool | `100` |
-| `HINDSIGHT_API_DB_COMMAND_TIMEOUT` | PostgreSQL command timeout in seconds | `60` |
+| `HINDSIGHT_API_DB_COMMAND_TIMEOUT` | PostgreSQL command timeout in seconds (asyncpg client-side) | `60` |
 | `HINDSIGHT_API_DB_ACQUIRE_TIMEOUT` | Connection acquisition timeout in seconds | `30` |
+| `HINDSIGHT_API_DB_STATEMENT_TIMEOUT` | Postgres `statement_timeout` applied to every pool connection, in seconds. Server-side safety net for runaway queries. Does **not** apply to Alembic migrations (which run on a separate psycopg2 engine). Set to `0` to disable. | `600` |
 
 For high-concurrency workloads, increase `DB_POOL_MAX_SIZE`. Each concurrent recall/think operation can use 2-4 connections.
 

skills/hindsight-docs/references/developer/configuration.md

@@ -45,8 +45,9 @@ Migrations will automatically create the schema if it doesn't exist and create a
 |----------|-------------|---------|
 | `HINDSIGHT_API_DB_POOL_MIN_SIZE` | Minimum connections in the pool | `5` |
 | `HINDSIGHT_API_DB_POOL_MAX_SIZE` | Maximum connections in the pool | `100` |
-| `HINDSIGHT_API_DB_COMMAND_TIMEOUT` | PostgreSQL command timeout in seconds | `60` |
+| `HINDSIGHT_API_DB_COMMAND_TIMEOUT` | PostgreSQL command timeout in seconds (asyncpg client-side) | `60` |
 | `HINDSIGHT_API_DB_ACQUIRE_TIMEOUT` | Connection acquisition timeout in seconds | `30` |
+| `HINDSIGHT_API_DB_STATEMENT_TIMEOUT` | Postgres `statement_timeout` applied to every pool connection, in seconds. Server-side safety net for runaway queries. Does **not** apply to Alembic migrations (which run on a separate psycopg2 engine). Set to `0` to disable. | `600` |
 
 For high-concurrency workloads, increase `DB_POOL_MAX_SIZE`. Each concurrent recall/think operation can use 2-4 connections.