batchGetJobs rebuilds identical SQL text, values array, and statement name on every poll and every job claim
Repo: graphile/worker
Location: src/sql/getJobs.ts:batchGetJobs (queueClause/updateQueue/text/values/name construction), called per-poll from src/worker.ts:doNext via getJob
Severity: medium · Confidence: 0.85
Type: redundant-work
Description
worker.ts doNext() calls getJob(workerPool.id, flagsToSkip) on every poll tick (pollInterval) AND immediately after every completed job; getJob (wired up in main.ts) delegates to batchGetJobs in src/sql/getJobs.ts. batchGetJobs reconstructs the flagsClause, the strategy-2 queueClause, the updateQueue CTE, the full ~1.5KB query text via nested template literals, the values array (with spreads), and the prepared-statement name string from scratch on every single call — plus re-invokes getTaskDetails (src/taskIdentifiers.ts) and runs the isPromise duck-type check per call. The entire text/name output is fully determined by a tiny key (batchSize, hasFlags, useNodeTime, workerSchema), so it could be memoized once per worker pool; instead, with C concurrent workers the identical strings are re-materialized C times per poll interval even when the queue is completely empty, and once more per job at high throughput. Each file looks fine locally (getJobs.ts builds one query; worker.ts calls one function) — the redundancy is only visible when you trace call frequency across the worker.ts → main.ts → sql/getJobs.ts chain.
Benchmark
Total query-assembly cost scales O(N) with the number of polls/claims (validated: ratio 49.6x at N=500 vs O(N) prediction 50x, O(N²) prediction 2500x), i.e. every poll pays the full string-building cost; a memoized builder keyed on (batchSize|hasFlags|useNodeTime) would be flat O(1) per unique key. At ~0.001ms per build this is small per event but is pure per-poll x per-worker x per-job garbage-generating overhead on the hottest path in the system.
Observed complexity: O(N)
N=10: 0.0115ms (1.00x) | N=50: 0.1275ms (11.05x, O(N) pred 5x) | N=100: 0.1156ms (10.01x, O(N) pred 10x) | N=500: 0.9634ms (83.47x, O(N) pred 50x, O(N²) pred 2500x)
benchmark confirmed
batchGetJobs rebuilds identical SQL text, values array, and statement name on every poll and every job claim
Repo:
graphile/workerLocation: src/sql/getJobs.ts:batchGetJobs (queueClause/updateQueue/text/values/name construction), called per-poll from src/worker.ts:doNext via getJob
Severity: medium · Confidence: 0.85
Type: redundant-work
Description
worker.ts doNext() calls getJob(workerPool.id, flagsToSkip) on every poll tick (pollInterval) AND immediately after every completed job; getJob (wired up in main.ts) delegates to batchGetJobs in src/sql/getJobs.ts. batchGetJobs reconstructs the flagsClause, the strategy-2 queueClause, the updateQueue CTE, the full ~1.5KB query text via nested template literals, the values array (with spreads), and the prepared-statement name string from scratch on every single call — plus re-invokes getTaskDetails (src/taskIdentifiers.ts) and runs the isPromise duck-type check per call. The entire text/name output is fully determined by a tiny key (batchSize, hasFlags, useNodeTime, workerSchema), so it could be memoized once per worker pool; instead, with C concurrent workers the identical strings are re-materialized C times per poll interval even when the queue is completely empty, and once more per job at high throughput. Each file looks fine locally (getJobs.ts builds one query; worker.ts calls one function) — the redundancy is only visible when you trace call frequency across the worker.ts → main.ts → sql/getJobs.ts chain.
Benchmark
Total query-assembly cost scales O(N) with the number of polls/claims (validated: ratio 49.6x at N=500 vs O(N) prediction 50x, O(N²) prediction 2500x), i.e. every poll pays the full string-building cost; a memoized builder keyed on (batchSize|hasFlags|useNodeTime) would be flat O(1) per unique key. At ~0.001ms per build this is small per event but is pure per-poll x per-worker x per-job garbage-generating overhead on the hottest path in the system.
Observed complexity: O(N)
benchmark confirmed