Skip to content

[LAB] Connector runtime audit – 2026-05-17 #102

Description

@NguyenCuong1989

Connector Runtime Audit – 2026-05-17

Architect: alpha_prime_omega Nguyễn Đức Cường
Executor: GPT-5.5 Thinking
Scope: 31 connected apps/connectors exposed through ChatGPT connector layer
Constraint: no financial-cost-generating action

Summary

This issue records a real connector lab run performed from ChatGPT using connected tools. The run moved beyond inventory and executed live read/auth/schema calls, then identified runtime mismatches and orchestration risks.

Verified connectors/actions

Connector Runtime result Notes
Airtable PARTIAL ping OK, but advertised mandatory get_additional_instructions returned This tool is not available.
Google Drive OK Auth/profile returned current Google identity.
Google Contacts OK Auth/profile returned same Google identity.
Gmail OK Label counts readable; mailbox scale is large.
Dropbox Legacy OK Profile readable; connector is still operational despite Legacy namespace.
GitHub OK Profile, repo listing, and DAIOF README fetch succeeded.
HubSpot OK with anomaly Permission matrix readable, but response serialized as JSON string in text.
Stripe READ ONLY OK Account and balance readable; account is livemode: true. No mutation performed.
Linear FAIL 401 Session expired. Please re-authenticate.
Notion OK with anomaly Current user readable, but response serialized as JSON string in text.

Defects / runtime anomalies

  1. LAB-RUNTIME-001 — Airtable mandatory tool unavailable

    • Registry advertises get_additional_instructions as mandatory.
    • Runtime call returns unavailable.
    • Impact: orchestration agents following tool instructions hit a hard failure before task execution.
  2. LAB-SERIALIZE-002 — inconsistent response envelope

    • HubSpot, Stripe, and Notion frequently return JSON inside text instead of structured result.
    • Impact: downstream parsers require double parsing and become connector-specific.
  3. LAB-AUTH-003 — stale auth exposed as available connector

    • Linear tools are visible, but runtime returns 401 session expired.
    • Impact: registry health does not equal runtime health.
  4. LAB-IDENTITY-004 — cross-connector identity fragmentation

    • Google/Dropbox/HubSpot/Notion identities use nguyencuong.2509@gmail.com.
    • GitHub identity uses nguyencuong.2509@icloud.com and handle NguyenCuong1989.
    • Impact: multi-connector automation needs an identity map, not email equality.
  5. LAB-SAFETY-005 — production financial connector exposure

    • Stripe balance retrieval showed livemode: true.
    • Impact: no automatic financial mutation should run unless sandbox/test mode is explicit.
  6. LAB-SCALE-006 — mailbox scale surface

    • Gmail has high unread volume; pagination and batch processing need stress tests.
  7. LAB-TOOLING-007 — wrong tool-name discovery failure

    • Attempted get_file_contents failed because GitHub connector exposes fetch_file instead.
    • Impact: tool discovery should include aliasing or better lookup by intent.

Recommended next actions

  • Normalize connector response envelopes to a common shape: {result, error, metadata}.
  • Add a registry health check layer: advertised, callable, authenticated, scoped, sandbox/live.
  • Add an identity map across Google, GitHub, HubSpot, Notion, Dropbox.
  • Keep Stripe/PayPal/DEWA under financial guardrails unless explicit sandbox evidence exists.
  • Build an automated connector smoke-test matrix under this repository.
  • Add regression test for Airtable mandatory preflight availability.
  • Re-authenticate Linear connector and rerun read-only tests.

D&R interpretation

Deconstruction: connector layer is broad and powerful, but runtime contracts are not uniform.
Focal point: the main bottleneck is not capability; it is operational consistency across auth, schema, serialization, and environment mode.
Re-architecture: use a connector supervisor that records health, identity, risk, and last-known-good operations before allowing higher-level autonomous workflows.

Short-term target

Create a repeatable Connector Lab harness that can run read-only smoke tests first, then controlled non-financial mutations in sandbox/lab resources.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions