|
| 1 | +# Oracle Database |
| 2 | + |
| 3 | +Hindsight uses PostgreSQL as its default storage backend, but it also runs on |
| 4 | +**Oracle Database 23ai** for organizations that standardize on Oracle |
| 5 | +infrastructure. All memory operations — retain, recall, and reflect — work the |
| 6 | +same way on Oracle; the backend is selected with a single environment variable. |
| 7 | + |
| 8 | +This guide covers everything needed to run Hindsight against Oracle: the |
| 9 | +prerequisites, the driver, a local quick start, provisioning a production |
| 10 | +database, running migrations, and the handful of behavioural differences from |
| 11 | +PostgreSQL. |
| 12 | + |
| 13 | +:::info When to use Oracle |
| 14 | +Oracle is the right choice when your organization already runs Oracle and needs |
| 15 | +Hindsight to live inside that footprint. For everything else, the default |
| 16 | +PostgreSQL backend is simpler to operate — see [Storage](./storage) for the |
| 17 | +rationale. Oracle and PostgreSQL are configured independently; you pick one per |
| 18 | +deployment. |
| 19 | +::: |
| 20 | + |
| 21 | +## Requirements |
| 22 | + |
| 23 | +| Requirement | Details | |
| 24 | +|-------------|---------| |
| 25 | +| Oracle Database | **23ai** (23.4+). [Oracle Database Free 23ai](https://www.oracle.com/database/free/) works for development. | |
| 26 | +| `VECTOR` type | Used for embeddings. Requires the schema to live in an **ASSM tablespace** (see below). | |
| 27 | +| Oracle Text | Full-text search uses Oracle Text indexes. The schema user needs the `CTXAPP` role. | |
| 28 | +| Driver | [`python-oracledb`](https://python-oracledb.readthedocs.io/) ≥ 2.5.0, running in **thin mode** — pure Python, no Oracle Instant Client required. | |
| 29 | + |
| 30 | +:::warning The schema must use an ASSM tablespace |
| 31 | +Oracle's `SYSTEM` tablespace uses *manual* segment space management (MSSM), |
| 32 | +which **does not support `VECTOR` columns**. Create the Hindsight user in a |
| 33 | +tablespace with **Automatic Segment Space Management (ASSM)** — otherwise |
| 34 | +migrations fail when they create embedding columns. The provisioning SQL below |
| 35 | +does this for you. |
| 36 | +::: |
| 37 | + |
| 38 | +## Install the driver |
| 39 | + |
| 40 | +The Oracle driver is an optional extra — it is not bundled with the default |
| 41 | +packages. Install it alongside Hindsight: |
| 42 | + |
| 43 | +```bash |
| 44 | +# With the packaged extra |
| 45 | +pip install "hindsight-api-slim[oracle]" |
| 46 | + |
| 47 | +# Or add the driver to an existing install (e.g. the full hindsight-api package) |
| 48 | +pip install hindsight-api oracledb |
| 49 | +``` |
| 50 | + |
| 51 | +If the driver is missing at startup, Hindsight fails with: |
| 52 | +`python-oracledb is required for Oracle backend. Install it with: pip install oracledb`. |
| 53 | + |
| 54 | +## Quick start (local Oracle) |
| 55 | + |
| 56 | +The fastest way to try Hindsight on Oracle is the bundled helper script, which |
| 57 | +starts a local **Oracle Database Free 23ai** container, provisions the test |
| 58 | +user with the correct tablespace and grants, and prints a ready-to-use |
| 59 | +connection URL: |
| 60 | + |
| 61 | +```bash |
| 62 | +# Start Oracle Free in Docker and bootstrap the hindsight_test user |
| 63 | +./scripts/dev/start-oracle.sh |
| 64 | + |
| 65 | +# ...prints: |
| 66 | +# export HINDSIGHT_API_DATABASE_BACKEND=oracle |
| 67 | +# export HINDSIGHT_API_DATABASE_URL='oracle+oracledb://hindsight_test:hindsight_test@localhost:1521/FREEPDB1' |
| 68 | + |
| 69 | +# Stop and remove the container when done |
| 70 | +./scripts/dev/stop-oracle.sh |
| 71 | +``` |
| 72 | + |
| 73 | +A cold start takes 60–120s while the database initializes. Once the script |
| 74 | +prints the connection URL, export the two variables it shows, run migrations, |
| 75 | +and start the API (see the steps below). This is the same setup Hindsight's CI |
| 76 | +uses to test the Oracle backend. |
| 77 | + |
| 78 | +## Production setup |
| 79 | + |
| 80 | +### 1. Provision the schema user |
| 81 | + |
| 82 | +Connect to your pluggable database as a privileged user (for example `SYSTEM`) |
| 83 | +and create a dedicated tablespace and user for Hindsight. The tablespace **must** |
| 84 | +use ASSM so `VECTOR` columns are supported: |
| 85 | + |
| 86 | +```sql |
| 87 | +-- ASSM tablespace (required for VECTOR columns). Size to your data volume. |
| 88 | +CREATE BIGFILE TABLESPACE hindsight_ts |
| 89 | + DATAFILE 'hindsight_ts.dbf' SIZE 2G AUTOEXTEND ON NEXT 500M MAXSIZE UNLIMITED |
| 90 | + EXTENT MANAGEMENT LOCAL |
| 91 | + SEGMENT SPACE MANAGEMENT AUTO; |
| 92 | + |
| 93 | +-- Dedicated schema user |
| 94 | +CREATE USER hindsight IDENTIFIED BY "<strong-password>" |
| 95 | + DEFAULT TABLESPACE hindsight_ts |
| 96 | + TEMPORARY TABLESPACE temp |
| 97 | + QUOTA UNLIMITED ON hindsight_ts; |
| 98 | + |
| 99 | +-- Object privileges Hindsight's migrations need |
| 100 | +GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE, CREATE VIEW, CREATE PROCEDURE TO hindsight; |
| 101 | + |
| 102 | +-- Oracle Text (full-text search indexes) |
| 103 | +GRANT CTXAPP TO hindsight; |
| 104 | +``` |
| 105 | + |
| 106 | +:::note Least privilege |
| 107 | +`CONNECT` and `RESOURCE` cover the basics; the explicit `CREATE TABLE / SEQUENCE |
| 108 | +/ VIEW / PROCEDURE` grants and `CTXAPP` are what the schema migrations require. |
| 109 | +No `DBA` role is needed. On a managed service where `CREATE TABLESPACE` is not |
| 110 | +available directly, provision the schema through the platform's admin tooling — |
| 111 | +the requirements are unchanged: an **ASSM** default tablespace (needed for |
| 112 | +`VECTOR` columns) plus the `CTXAPP` role. |
| 113 | +::: |
| 114 | + |
| 115 | +### 2. Build the connection URL |
| 116 | + |
| 117 | +Hindsight uses SQLAlchemy-style URLs. The Oracle form is: |
| 118 | + |
| 119 | +``` |
| 120 | +oracle+oracledb://USER:PASSWORD@HOST:PORT/SERVICE_NAME |
| 121 | +``` |
| 122 | + |
| 123 | +| Part | Example | Notes | |
| 124 | +|------|---------|-------| |
| 125 | +| `USER` / `PASSWORD` | `hindsight` / `s3cret` | The schema user from step 1. URL-encode reserved characters (`@`, `/`, `:`) in the password. | |
| 126 | +| `HOST:PORT` | `db.internal:1521` | The listener host and port (Oracle default is `1521`). | |
| 127 | +| `SERVICE_NAME` | `FREEPDB1` | The **service name** of your pluggable database (not the SID). `FREEPDB1` for Oracle Free. | |
| 128 | + |
| 129 | +Example: |
| 130 | + |
| 131 | +``` |
| 132 | +oracle+oracledb://hindsight:s3cret@db.internal:1521/ORCLPDB1 |
| 133 | +``` |
| 134 | + |
| 135 | +:::warning Connection support: Easy Connect only |
| 136 | +Hindsight builds the Oracle connection from the URL as a plain |
| 137 | +`host:port/service_name` descriptor. **Wallet-based mTLS, TLS/TCPS, and TNS |
| 138 | +aliases or full connect descriptors are not currently supported** by the |
| 139 | +connection layer. In practice: |
| 140 | + |
| 141 | +- **Oracle Autonomous Database** and other services that require a wallet / |
| 142 | + mTLS are not supported as-is — connect to a database reachable over a direct |
| 143 | + `host:port/service` listener. |
| 144 | +- The driver does not negotiate TLS itself, so secure the connection at the |
| 145 | + network layer (private networking, VPN, or a TLS-terminating proxy). |
| 146 | +::: |
| 147 | + |
| 148 | +### 3. Configure Hindsight |
| 149 | + |
| 150 | +Point Hindsight at Oracle with two environment variables: |
| 151 | + |
| 152 | +```bash |
| 153 | +export HINDSIGHT_API_DATABASE_BACKEND=oracle |
| 154 | +export HINDSIGHT_API_DATABASE_URL='oracle+oracledb://hindsight:s3cret@db.internal:1521/ORCLPDB1' |
| 155 | +``` |
| 156 | + |
| 157 | +`HINDSIGHT_API_DATABASE_BACKEND` defaults to `postgresql`; set it to `oracle` to |
| 158 | +select the Oracle backend. See [Configuration → Database](./configuration#database) |
| 159 | +for the full list of database variables. |
| 160 | + |
| 161 | +### 4. Run migrations |
| 162 | + |
| 163 | +Hindsight runs the same schema migrations on Oracle as on PostgreSQL. By default |
| 164 | +the API applies them automatically on startup |
| 165 | +(`HINDSIGHT_API_RUN_MIGRATIONS_ON_STARTUP=true`). To run them explicitly — for |
| 166 | +example in a controlled deploy step — use: |
| 167 | + |
| 168 | +```bash |
| 169 | +hindsight-admin run-db-migration |
| 170 | +``` |
| 171 | + |
| 172 | +This routes through the dialect-aware migration runner and creates the Oracle |
| 173 | +schema. (Unlike the admin CLI's data-movement commands, `run-db-migration` |
| 174 | +is fully supported on Oracle — see [Limitations](#limitations-vs-postgresql).) |
| 175 | + |
| 176 | +### 5. Start the API |
| 177 | + |
| 178 | +```bash |
| 179 | +hindsight-api |
| 180 | +``` |
| 181 | + |
| 182 | +On startup Hindsight logs the resolved database (with credentials masked); it |
| 183 | +should show your Oracle host and confirm the Oracle backend is active. |
| 184 | + |
| 185 | +## Configuration reference |
| 186 | + |
| 187 | +Oracle-relevant settings, all documented in full on the |
| 188 | +[Configuration](./configuration) page: |
| 189 | + |
| 190 | +| Variable | Purpose | |
| 191 | +|----------|---------| |
| 192 | +| `HINDSIGHT_API_DATABASE_BACKEND` | `postgresql` (default) or `oracle`. | |
| 193 | +| `HINDSIGHT_API_DATABASE_URL` | `oracle+oracledb://…` connection URL. | |
| 194 | +| `HINDSIGHT_API_RUN_MIGRATIONS_ON_STARTUP` | Auto-apply migrations when the API boots (default `true`). | |
| 195 | + |
| 196 | +## Limitations vs PostgreSQL |
| 197 | + |
| 198 | +Memory operations behave identically on Oracle, but a few operational and |
| 199 | +internal details differ: |
| 200 | + |
| 201 | +- **Admin CLI data commands are PostgreSQL-only.** `hindsight-admin` backup, |
| 202 | + restore, bank export/import, and worker-status use asyncpg binary `COPY` and |
| 203 | + `TRUNCATE`, which are PostgreSQL-specific and not available on Oracle. |
| 204 | + Schema migrations (`run-db-migration`) *are* supported on Oracle. |
| 205 | +- **No embedded database.** The `pg0` embedded PostgreSQL used for zero-config |
| 206 | + local development has no Oracle equivalent — Oracle always requires a running |
| 207 | + instance (use the [quick-start script](#quick-start-local-oracle) locally). |
| 208 | +- **Consolidation reconciliation is skipped.** The similarity-based |
| 209 | + near-duplicate reconciliation pass in consolidation |
| 210 | + (`HINDSIGHT_API_CONSOLIDATION_DEDUP_THRESHOLD`) is a PostgreSQL-only path; |
| 211 | + consolidation still runs on Oracle, without that extra reconciliation step. |
| 212 | +- **Entity resolution uses Oracle fuzzy matching.** Fuzzy entity lookup during |
| 213 | + retain uses Oracle's text matching rather than PostgreSQL's `pg_trgm` trigram |
| 214 | + matching. Behaviour is equivalent; the underlying mechanism differs. |
| 215 | + |
| 216 | +## Troubleshooting |
| 217 | + |
| 218 | +| Symptom | Cause / Fix | |
| 219 | +|---------|-------------| |
| 220 | +| `python-oracledb is required for Oracle backend` | The driver isn't installed. Run `pip install oracledb` (or install the `[oracle]` extra). | |
| 221 | +| Migration errors when creating embedding/`VECTOR` columns | The schema user's default tablespace is not ASSM (often the `SYSTEM` tablespace). Recreate the user in an ASSM tablespace as shown above. | |
| 222 | +| Full-text search errors / missing Oracle Text index | The schema user is missing the `CTXAPP` role. Run `GRANT CTXAPP TO <user>;`. | |
| 223 | +| `ORA-12514` / service not found | The URL uses a SID or wrong service name. Use the pluggable database **service name** (e.g. `FREEPDB1`), not the SID. | |
| 224 | +| Login works manually but fails from Hindsight | A reserved character in the password isn't URL-encoded. Encode `@ / : ?` in the `DATABASE_URL`. | |
| 225 | + |
| 226 | +## See also |
| 227 | + |
| 228 | +- [Storage](./storage) — why PostgreSQL is the default, and how Oracle fits in |
| 229 | +- [Configuration](./configuration#database) — all database environment variables |
| 230 | +- [Installation](./installation) — packaging and deployment options |
| 231 | +- [Admin CLI](./admin-cli) — administrative commands (PostgreSQL-only data operations) |
0 commit comments