Embed testId in Veo/Grok video poll ids for multi-tenant isolation (#278) (#286)

## What

Completes the #278 async-video providers (Veo, Grok) for **multi-tenant
use**: each provider now embeds the `testId` into the poll/operation id
it returns, so a poll that carries the testId only in the URL (not in a
header) still resolves to the correct tenant's in-flight job. Mirrors
the existing OpenRouter `polling_url` testId-embedding pattern.

No version bump — stays `1.34.0`; the dedicated `chore: release v1.35.0`
PR owns the bump.

## Why

When the original #278 video work landed, the submit→poll round-trip
relied on the testId arriving via header on the poll. Real provider SDKs
poll the returned operation/job URL directly and don't necessarily
re-send a custom header, so under concurrent multi-tenant use a
header-less poll fell to the default testId → key miss → 404 (the job
was stored under `${testId}:${id}`). Embedding the testId in the
returned id makes it travel with the poll URL.

## Changes

- `src/veo-video.ts` / `src/grok-video.ts`: append `?testId=<encoded>`
to the returned poll/operation id (only when a non-default testId is
present — single-tenant flows still return a bare id). The router strips
the query before route-matching; `getTestId` re-derives the testId from
the query when no header is present.
- Tests: a decisive cross-tenant routing test in both suites — two
tenants submit jobs concurrently, and each tenant's **header-less** poll
of its returned suffix-bearing id resolves **its own** job (asserts the
specific payload, not just "not 404"). Replaces an earlier test that was
tautological (passed even pre-fix). Suffix is asserted with an anchored
`?testId=<encoded>` regex; back-compat (default testId → bare id)
retained.

## Test plan

- Full suite: **144 files / 4250 tests** pass; `tsc`, `eslint`,
`prettier --check`, `build`, `test:exports`
(node10/node16-CJS/ESM/bundler) all green.
- Red-green proven on the real submit+poll surface: reverting only
`veo-video.ts`/`grok-video.ts` makes both concurrent-tenant tests fail
(header-less poll 404s); restoring passes 31/31.

## Notes

No cross-tenant data-leak existed pre-fix (a missing/mismatched testId
yields a key miss → loud 404, never a tenant-agnostic scan); this change
is about header-less polls resolving **correctly** per tenant, confirmed
by an independent silent-failure review.

Author: jpr5

src/__tests__/grok-video.test.ts

@@ -363,6 +363,123 @@ describe("GET /v1/videos/{id} dispatch journals handler errors as 500 (Grok-firs
   });
 });
 
+// ─── Multi-tenant poll isolation (#278) ──────────────────────────────────────
+// Submit stores the job keyed `${testId}:${requestId}`, but the returned
+// request_id is opaque. A multi-tenant client polls the returned id WITHOUT an
+// x-test-id header, so the testId must travel in the returned id (via
+// testIdSuffix) for getTestId's `?testId=` fallback to resolve the scope —
+// mirroring OpenRouter's polling_url treatment.
+
+describe("Grok video — testId scoping of returned request_id", () => {
+  let mock: LLMock | undefined;
+
+  afterEach(async () => {
+    await mock?.stop();
+    mock = undefined;
+  });
+
+  test("returned request_id carries testId and resolves the poll without the header", async () => {
+    mock = new LLMock({ port: 0 });
+    mock.addFixture({
+      match: { userMessage: "scoped poll", endpoint: "video" },
+      response: { video: { id: "vid_scoped", status: "completed", url: "https://cdn.x.ai/s.mp4" } },
+    });
+    await mock.start();
+
+    const submit = await fetch(`${mock.url}/v1/videos/generations`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json", "X-Test-Id": "test-a" },
+      body: JSON.stringify({ model: "grok-imagine-video", prompt: "scoped poll" }),
+    });
+    const { request_id } = (await submit.json()) as { request_id: string };
+    // Exact query param, not just a substring: `{uuid}?testId=test-a`.
+    expect(request_id).toMatch(/^[^?]+\?testId=test-a$/);
+
+    // Bare poll of the returned id — no X-Test-Id header — must still resolve
+    // via the embedded query parameter.
+    const poll = await fetch(`${mock.url}/v1/videos/${request_id}`);
+    expect(poll.status).toBe(200);
+    const data = await poll.json();
+    expect(data.status).toBe("done");
+  });
+
+  // DECISIVE multi-tenant routing (#278). Two tenants submit DISTINCT jobs
+  // concurrently — both live in the map at once — then each polls ONLY its own
+  // RETURNED suffix-bearing request_id, sending NO x-test-id header. The
+  // header-less poll must resolve via the embedded `?testId=` suffix to that
+  // tenant's own job (not the other tenant's, not a Sora 404 fall-through).
+  // This FAILS on pre-fix code: a bare returned request_id carries no testId, so
+  // a header-less poll falls to the default testId, the `${testId}:${requestId}`
+  // key misses, and the Grok-first dispatch falls through to Sora → 404. The
+  // earlier header-stripping "isolation" test was tautological (it 404s pre- AND
+  // post-fix); this one is the real cross-tenant routing proof.
+  test("concurrent tenants each resolve their OWN job via the header-less suffix", async () => {
+    mock = new LLMock({ port: 0 });
+    mock.addFixture({
+      match: { userMessage: "tenant A clip", endpoint: "video" },
+      response: { video: { id: "vid_A", status: "completed", url: "https://cdn.x.ai/a.mp4" } },
+    });
+    mock.addFixture({
+      match: { userMessage: "tenant B clip", endpoint: "video" },
+      response: { video: { id: "vid_B", status: "completed", url: "https://cdn.x.ai/b.mp4" } },
+    });
+    await mock.start();
+
+    // Both tenants submit concurrently → both jobs coexist in the map.
+    const [resA, resB] = await Promise.all([
+      fetch(`${mock.url}/v1/videos/generations`, {
+        method: "POST",
+        headers: { "Content-Type": "application/json", "X-Test-Id": "tenant-a" },
+        body: JSON.stringify({ model: "grok-imagine-video", prompt: "tenant A clip" }),
+      }),
+      fetch(`${mock.url}/v1/videos/generations`, {
+        method: "POST",
+        headers: { "Content-Type": "application/json", "X-Test-Id": "tenant-b" },
+        body: JSON.stringify({ model: "grok-imagine-video", prompt: "tenant B clip" }),
+      }),
+    ]);
+    const { request_id: idA } = (await resA.json()) as { request_id: string };
+    const { request_id: idB } = (await resB.json()) as { request_id: string };
+    expect(idA).toMatch(/^[^?]+\?testId=tenant-a$/);
+    expect(idB).toMatch(/^[^?]+\?testId=tenant-b$/);
+    // Distinct underlying request ids (no accidental collision).
+    expect(idA.split("?")[0]).not.toBe(idB.split("?")[0]);
+
+    // Header-less poll of A's returned id → resolves A's job (a.mp4), not B's.
+    const pollA = await fetch(`${mock.url}/v1/videos/${idA}`);
+    expect(pollA.status).toBe(200);
+    const dataA = await pollA.json();
+    expect(dataA.status).toBe("done");
+    expect(dataA.video.url).toBe("https://cdn.x.ai/a.mp4");
+
+    // Header-less poll of B's returned id → resolves B's job (b.mp4), not A's.
+    const pollB = await fetch(`${mock.url}/v1/videos/${idB}`);
+    expect(pollB.status).toBe(200);
+    const dataB = await pollB.json();
+    expect(dataB.status).toBe("done");
+    expect(dataB.video.url).toBe("https://cdn.x.ai/b.mp4");
+  });
+
+  test("default testId returns the bare request_id (no testId param)", async () => {
+    mock = new LLMock({ port: 0 });
+    mock.addFixture({
+      match: { userMessage: "clean id", endpoint: "video" },
+      response: { video: { id: "vid_clean", status: "completed", url: "https://cdn.x.ai/c.mp4" } },
+    });
+    await mock.start();
+
+    const submit = await fetch(`${mock.url}/v1/videos/generations`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ model: "grok-imagine-video", prompt: "clean id" }),
+    });
+    const { request_id } = (await submit.json()) as { request_id: string };
+    // Byte-identical to pre-fix behaviour: bare uuid, no query.
+    expect(request_id).not.toContain("testId=");
+    expect(request_id).not.toContain("?");
+  });
+});
+
 // Type-only smoke: the stored video status union remains the 3-value set
 // (no "done"); the Grok wire "done" is derived at serialization.
 describe("VideoResponse stored status", () => {

src/__tests__/veo-video.test.ts

@@ -278,3 +278,132 @@ describe("GET /v1beta/operations/{name} dispatch journals handler errors as 500
     expect(failed[0].method).toBe("GET");
   });
 });
+
+// ─── Multi-tenant poll isolation (#278) ──────────────────────────────────────
+// Submit stores the job keyed `${testId}:${operationName}`, but the returned
+// operation name is opaque. A multi-tenant client polls the returned name
+// WITHOUT an x-test-id header, so the testId must travel in the returned name
+// (via testIdSuffix) for getTestId's `?testId=` fallback to resolve the scope —
+// mirroring OpenRouter's polling_url treatment.
+
+describe("Veo video — testId scoping of returned operation name", () => {
+  let mock: LLMock | undefined;
+
+  afterEach(async () => {
+    await mock?.stop();
+    mock = undefined;
+  });
+
+  test("returned name carries testId and resolves the poll without the header", async () => {
+    mock = new LLMock({ port: 0 });
+    mock.addFixture({
+      match: { userMessage: "scoped poll", endpoint: "video" },
+      response: {
+        video: { id: "veo_scoped", status: "completed", url: "https://files.example/s.mp4" },
+      },
+    });
+    await mock.start();
+
+    const submit = await fetch(submitUrl(mock.url), {
+      method: "POST",
+      headers: { "Content-Type": "application/json", "X-Test-Id": "test-a" },
+      body: JSON.stringify({ instances: [{ prompt: "scoped poll" }] }),
+    });
+    const { name } = (await submit.json()) as { name: string };
+    // Exact query param, not just a substring: `operations/{uuid}?testId=test-a`.
+    expect(name).toMatch(/^operations\/[^?]+\?testId=test-a$/);
+
+    // Bare poll of the returned name — no X-Test-Id header — must still resolve
+    // via the embedded query parameter.
+    const poll = await fetch(`${mock.url}/v1beta/${name}`);
+    expect(poll.status).toBe(200);
+    const data = await poll.json();
+    expect(data.done).toBe(true);
+  });
+
+  // DECISIVE multi-tenant routing (#278). Two tenants submit DISTINCT jobs
+  // concurrently — both live in the map at once — then each polls ONLY its own
+  // RETURNED suffix-bearing id, sending NO x-test-id header. The header-less
+  // poll must resolve via the embedded `?testId=` suffix to that tenant's own
+  // job (not the other tenant's, not 404). This FAILS on pre-fix code: a bare
+  // returned name carries no testId, so a header-less poll falls to the default
+  // testId and the `${testId}:${operationName}` key misses → 404. The earlier
+  // header-stripping "isolation" test was tautological (it 404s pre- AND
+  // post-fix); this one is the real cross-tenant routing proof.
+  test("concurrent tenants each resolve their OWN job via the header-less suffix", async () => {
+    mock = new LLMock({ port: 0 });
+    mock.addFixture({
+      match: { userMessage: "tenant A clip", endpoint: "video" },
+      response: {
+        video: { id: "veo_A", status: "completed", url: "https://files.example/a.mp4" },
+      },
+    });
+    mock.addFixture({
+      match: { userMessage: "tenant B clip", endpoint: "video" },
+      response: {
+        video: { id: "veo_B", status: "completed", url: "https://files.example/b.mp4" },
+      },
+    });
+    await mock.start();
+
+    // Both tenants submit concurrently → both jobs coexist in the map.
+    const [resA, resB] = await Promise.all([
+      fetch(submitUrl(mock.url), {
+        method: "POST",
+        headers: { "Content-Type": "application/json", "X-Test-Id": "tenant-a" },
+        body: JSON.stringify({ instances: [{ prompt: "tenant A clip" }] }),
+      }),
+      fetch(submitUrl(mock.url), {
+        method: "POST",
+        headers: { "Content-Type": "application/json", "X-Test-Id": "tenant-b" },
+        body: JSON.stringify({ instances: [{ prompt: "tenant B clip" }] }),
+      }),
+    ]);
+    const { name: nameA } = (await resA.json()) as { name: string };
+    const { name: nameB } = (await resB.json()) as { name: string };
+    expect(nameA).toMatch(/^operations\/[^?]+\?testId=tenant-a$/);
+    expect(nameB).toMatch(/^operations\/[^?]+\?testId=tenant-b$/);
+    // Distinct underlying operation names (no accidental collision).
+    expect(nameA.split("?")[0]).not.toBe(nameB.split("?")[0]);
+
+    // Header-less poll of A's returned name → resolves A's job (a.mp4), not B's.
+    const pollA = await fetch(`${mock.url}/v1beta/${nameA}`);
+    expect(pollA.status).toBe(200);
+    const dataA = await pollA.json();
+    expect(dataA.done).toBe(true);
+    expect(dataA.response.generateVideoResponse.generatedSamples[0].video.uri).toBe(
+      "https://files.example/a.mp4",
+    );
+
+    // Header-less poll of B's returned name → resolves B's job (b.mp4), not A's.
+    const pollB = await fetch(`${mock.url}/v1beta/${nameB}`);
+    expect(pollB.status).toBe(200);
+    const dataB = await pollB.json();
+    expect(dataB.done).toBe(true);
+    expect(dataB.response.generateVideoResponse.generatedSamples[0].video.uri).toBe(
+      "https://files.example/b.mp4",
+    );
+  });
+
+  test("default testId returns the bare operation name (no testId param)", async () => {
+    mock = new LLMock({ port: 0 });
+    mock.addFixture({
+      match: { userMessage: "clean name", endpoint: "video" },
+      response: {
+        video: { id: "veo_clean", status: "completed", url: "https://files.example/c.mp4" },
+      },
+    });
+    await mock.start();
+
+    const submit = await fetch(submitUrl(mock.url), {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ instances: [{ prompt: "clean name" }] }),
+    });
+    const { name } = (await submit.json()) as { name: string };
+    // Byte-identical to pre-fix behaviour: bare `operations/{uuid}`, no query.
+    expect(name).not.toContain("testId=");
+    expect(name).not.toContain("?");
+    expect(name.startsWith("operations/")).toBe(true);
+  });
+});

src/grok-video.ts

@@ -33,7 +33,7 @@ import {
 } from "./recorder.js";
 import { resolveUpstreamUrl } from "./url.js";
 import { handleVideoStatus, type VideoStateMap } from "./video.js";
-import { readEnvelopeText, upstreamTimeoutSignal } from "./video-proxy-shared.js";
+import { readEnvelopeText, testIdSuffix, upstreamTimeoutSignal } from "./video-proxy-shared.js";
 
 /**
  * xAI Grok Imagine async video lifecycle mock. Submit
@@ -555,8 +555,12 @@ export async function handleGrokVideoCreate(
     );
   }
 
+  // Embed the testId in the RETURNED request_id (not the stored key) so a
+  // multi-tenant client can poll the opaque id without an x-test-id header — the
+  // testId travels via getTestId's `?testId=` fallback. Mirrors OpenRouter's
+  // polling_url treatment; testIdSuffix is "" for the default testId (bare id).
   res.writeHead(200, { "Content-Type": "application/json" });
-  res.end(JSON.stringify({ request_id: requestId }));
+  res.end(JSON.stringify({ request_id: requestId + testIdSuffix(testId, "?") }));
 }
 
 // ─── GET /v1/videos/{id} — status poll (Grok-first, Sora fall-through) ───────
@@ -872,8 +876,11 @@ async function proxyGrokVideoSubmit(args: {
     },
   });
 
+  // Embed the testId in the RETURNED request_id (see the replay submit) so
+  // record-mode multi-tenant polls resolve via `?testId=`; the upstream poll
+  // still uses the bare job.requestId / job.upstreamPollingUrl, untouched.
   res.writeHead(200, { "Content-Type": "application/json" });
-  res.end(JSON.stringify({ request_id: requestId }));
+  res.end(JSON.stringify({ request_id: requestId + testIdSuffix(testId, "?") }));
   return "handled";
 }
 

src/veo-video.ts

@@ -32,7 +32,7 @@ import {
   sanitizeHeaderValue,
 } from "./recorder.js";
 import { resolveUpstreamUrl } from "./url.js";
-import { readEnvelopeText, upstreamTimeoutSignal } from "./video-proxy-shared.js";
+import { readEnvelopeText, testIdSuffix, upstreamTimeoutSignal } from "./video-proxy-shared.js";
 
 /**
  * Google Veo async video lifecycle mock. Submit
@@ -461,8 +461,12 @@ export async function handleVeoVideoCreate(
     );
   }
 
+  // Embed the testId in the RETURNED name (not the stored key) so a multi-tenant
+  // client can poll the opaque name without an x-test-id header — the testId
+  // travels via getTestId's `?testId=` fallback. Mirrors OpenRouter's polling_url
+  // treatment; testIdSuffix is "" for the default testId, keeping the name bare.
   res.writeHead(200, { "Content-Type": "application/json" });
-  res.end(JSON.stringify({ name: operationName }));
+  res.end(JSON.stringify({ name: operationName + testIdSuffix(testId, "?") }));
 }
 
 // ─── GET /v1beta/operations/{name} — status poll ─────────────────────────────
@@ -801,8 +805,11 @@ async function proxyVeoVideoSubmit(args: {
     },
   });
 
+  // Embed the testId in the RETURNED name (see the replay submit) so record-mode
+  // multi-tenant polls resolve via `?testId=`; the upstream poll still uses the
+  // bare job.operationName / job.upstreamPollingUrl, untouched by the suffix.
   res.writeHead(200, { "Content-Type": "application/json" });
-  res.end(JSON.stringify({ name: operationName }));
+  res.end(JSON.stringify({ name: operationName + testIdSuffix(testId, "?") }));
   return "handled";
 }