Skip to content

connectors.md

Latest commit

 

History

History
382 lines (280 loc) · 11 KB

connectors.md

File metadata and controls

382 lines (280 loc) · 11 KB
Raw

Connectors CLI Reference

remnic connectors is the operator surface for inspecting and manually controlling the live connectors that continuously ingest content from external services (Google Drive, Notion, Gmail, GitHub, …) into your memory directory.

This page documents configuration keys, environment variables, OAuth setup notes per connector, the full CLI reference, and troubleshooting. For the framework contract (how to write a connector) see live-connectors.md.

Quick reference

# List all configured connectors (human-readable table)
remnic connectors list

# Machine-readable JSON status (useful in scripts / CI)
remnic connectors status

# Manually run one incremental sync (operator debug)
remnic connectors run google-drive
remnic connectors run notion

Subcommand: remnic connectors list

Lists every configured live connector with its enabled state, last poll time, cumulative docs imported, and last error (if any).

Usage: remnic connectors list [options]

Options:
  --format <fmt>   Output format: text (default), markdown, or json

Example — text output (default):

Live connectors (2):

  google-drive  (Google Drive)
    state:         enabled, ok
    last_poll:     2026-04-25T08:00:00.000Z
    docs_imported: 137

  notion  (Notion)
    state:         enabled, error
    last_poll:     2026-04-25T09:00:00.000Z
    docs_imported: 0
    last_error:    invalid_token: The token you provided is invalid.

Example — --format json:

[
  {
    "id": "google-drive",
    "displayName": "Google Drive",
    "enabled": true,
    "lastSyncAt": "2026-04-25T08:00:00.000Z",
    "lastSyncStatus": "success",
    "lastSyncError": null,
    "totalDocsImported": 137,
    "updatedAt": "2026-04-25T08:00:00.000Z"
  }
]

Subcommand: remnic connectors status

Identical data to list but defaults to JSON output so shell scripts can reliably jq-parse the result without requiring --format json every time.

Usage: remnic connectors status [options]

Options:
  --format <fmt>   Output format: json (default), text, or markdown

Subcommand: remnic connectors run <name>

Manually triggers one incremental syncIncremental() pass for the named connector. Useful when:

  • You want to verify credentials work without waiting for the next scheduler tick.
  • A sync failed and you want to retry immediately.
  • You are debugging cursor advancement.
Usage: remnic connectors run <name> [options]

Arguments:
  name             Connector id (e.g. google-drive, notion)

Options:
  --format <fmt>   Output format: text (default), markdown, or json

On success, exits 0 and prints the number of new documents imported. On failure, exits 1, writes the error to stderr, and records the failure in the connector's state file so connectors list reflects it.

The maintenance scheduler calls the MCP tool engram.live_connectors_run every minute once connectors are configured through the engram-live-connectors-sync cron. That runner honors each connector's pollIntervalMs and covers every enabled built-in connector. remnic connectors run <name> currently supports Google Drive and Notion as the single-connector debug path.

Example — success:

connectors run: google-drive — OK
  docs_imported: 5

Example — failure (stderr, exit 1):

connectors run: notion — FAILED
  docs_imported: 0
  error:         invalid_token: The token you provided is invalid.

Connector: Google Drive

Prerequisites

  1. Create an OAuth2 client in Google Cloud Console.
    • Application type: Web application (or Desktop app for personal use).
    • Add an authorised redirect URI: http://localhost:8080/callback (or your preferred redirect).
  2. Enable the Google Drive API in the project.
  3. Obtain a refresh token using the OAuth2 flow with scope https://www.googleapis.com/auth/drive.readonly. Tools like oauth2l or a small helper script can drive the consent flow and print the refresh token.

Config keys

// In your Remnic config (e.g. ~/.config/engram/config.json)
{
  "connectors": {
    "googleDrive": {
      "enabled": true,
      "clientId": "${GOOGLE_DRIVE_CLIENT_ID}",
      "clientSecret": "${GOOGLE_DRIVE_CLIENT_SECRET}",
      "refreshToken": "${GOOGLE_DRIVE_REFRESH_TOKEN}",
      "pollIntervalMs": 300000,
      "folderIds": []
    }
  }
}
Key Default Description
enabled false Master gate. Set to true to activate the connector.
clientId "" OAuth2 client ID from Google Cloud Console.
clientSecret "" OAuth2 client secret.
refreshToken "" Long-lived refresh token obtained from the OAuth2 consent flow.
pollIntervalMs 300000 Poll interval in milliseconds. Min 1000 (1 s), max 86400000 (24 h).
folderIds [] Limit import to files whose Google Drive parents intersect this list. Empty = all accessible files.

Environment variables

Config values can be populated from environment variables by using ${VAR_NAME} placeholders in your config file if your config layer supports interpolation, or by setting the values directly from a secrets manager / systemd EnvironmentFile at startup:

Variable Maps to
GOOGLE_DRIVE_CLIENT_ID connectors.googleDrive.clientId
GOOGLE_DRIVE_CLIENT_SECRET connectors.googleDrive.clientSecret
GOOGLE_DRIVE_REFRESH_TOKEN connectors.googleDrive.refreshToken

Never commit real credential values. Use placeholders in checked-in config files.

Optional npm dependency

The googleapis package is not bundled with @remnic/core. Install it only if you enable this connector:

npm install googleapis
# or
pnpm add googleapis

The connector loads it via a computed-specifier dynamic import so bundlers cannot include it accidentally and the package is not downloaded by users who never configure Drive.

What is imported

  • Google Docs / Sheets / Slides: exported to plain text via files.export.
  • Plain-text MIME types (text/plain, text/markdown, etc.): downloaded directly via files.get?alt=media.
  • Everything else (images, PDFs, archives, binary files): skipped. Binary files belong in the binary-lifecycle pipeline, not the textual ingestion path.

The first sync seeds the Drive startPageToken without importing any files. Subsequent syncs only pull files changed since the previous token.

Connector: Notion

Prerequisites

  1. Go to https://www.notion.so/my-integrations and create a new integration.
  2. Give it a name (e.g. "Remnic") and select the workspace.
  3. Set Capabilities to at minimum Read content.
  4. Copy the Internal Integration Token.
  5. In each Notion database you want to import, open Share and invite your integration.

Config keys

{
  "connectors": {
    "notion": {
      "enabled": true,
      "token": "${NOTION_INTEGRATION_TOKEN}",
      "databaseIds": ["<your-database-id>"],
      "pollIntervalMs": 300000
    }
  }
}
Key Default Description
enabled false Master gate.
token "" Notion Internal Integration Token.
databaseIds [] List of Notion database IDs to import. Empty = connector does nothing.
pollIntervalMs 300000 Poll interval in milliseconds. Min 1000, max 86400000.

Environment variables

Variable Maps to
NOTION_INTEGRATION_TOKEN connectors.notion.token

How to find a database ID

From a Notion database URL:

https://www.notion.so/myworkspace/My-Database-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx?v=...

The database ID is the 32-character hex string between the last / and ?.

What is imported

  • Database pages: all pages in the configured databases whose last_edited_time is after the per-page high-water mark in the cursor.
  • Page content: text extracted from supported block types (paragraph, heading, bulleted/numbered list, toggle, quote, code, callout). Unsupported block types (images, embeds, databases) are silently skipped.
  • The first sync seeds the watermark from the current state of each database without importing any content.

State files

Per-connector state (cursor + sync metadata) lives at:

<memoryDir>/state/connectors/<connector-id>.json

These files travel with your memory directory. Back them up along with the rest of <memoryDir>. To reset a connector's cursor (force a re-sync from the beginning), delete the corresponding .json file — the next sync will seed a fresh cursor.

Do not edit state files by hand while a sync is running — writes are atomic (temp + rename) but a manual edit during an active sync could race.

Troubleshooting

connectors list shows all connectors as disabled

Connectors are opt-in. Set connectors.<name>.enabled: true in your config and ensure credentials are populated.

Google Drive: googleapis package not found

Install the optional dependency:

npm install googleapis

Restart the daemon (launchctl kickstart -k gui/501/ai.openclaw.gateway on macOS, or your systemd unit) after installing.

Google Drive: invalid_grant on refresh

The refresh token has expired or been revoked (e.g., by a Google account password change or explicit revocation in https://myaccount.google.com/permissions).

Re-run the OAuth2 consent flow to obtain a new refresh token, then update connectors.googleDrive.refreshToken in your config.

Notion: unauthorized / token errors

Verify the integration token is correct:

curl -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
     -H "Notion-Version: 2022-06-28" \
     https://api.notion.com/v1/users/me

Also confirm the integration has been invited to each database you listed in databaseIds.

Notion: database returns no pages

Check that databaseIds contains the correct database ID (32-character hex string, no hyphens in the raw API form). Also confirm the integration has Read content capability and has been added to the database via the Share menu.

connectors run exits 1 immediately

The connector may be disabled in config (enabled: false). Check with:

remnic connectors status

Cursor is stuck / connector keeps re-ingesting

Delete the state file to reset:

rm <memoryDir>/state/connectors/<connector-id>.json
remnic connectors run <connector-id>  # re-seeds cursor

The first run after deletion emits zero documents (cursor seed pass). Subsequent runs pick up new content.

See also