Stabilize EVM trace timeout tests with deterministic unit coverage

[Unique-Divine]

Integration test file eth/rpc/rpcapi/tracing_test.go exercises debug trace RPCs against a live localnet. One case in method TestTraceBlock sets TraceConfig.Timeout = "1ns" to force tracer timeout during debug_traceBlockByNumber. That case is marked mayTimeoutRace because the outcome depends on scheduler timing and machine load—not on a controllable execution path.

Problem

The ultra-small timeout case does not provide deterministic signal about timeout semantics in function TraceEthTxMsg (file evm/evmstate/grpc_query.go). Under load, the same test can pass with a successful trace, fail with "execution timeout", or fail with unrelated errors. That makes CI flaky and obscures whether timeout handling actually regressed.

Function TraceEthTxMsg already documents a known race: a very small timeout can stop geth's callTracer before it initializes its root call frame, which may panic during EVM callbacks or result collection. A recover block maps deadline-exceeded panics to gRPC code DeadlineExceeded with message "execution timeout", but other panics surface as internal errors. Default timeout constant DefaultGethTraceTimeout is five seconds; the test's one-nanosecond override sits far outside normal operating conditions.

Tightening the integration assertion to accept only success or "execution timeout" exposed a third outcome during focused runs:

rpc error: code = Unknown desc = runtime error: index out of range [0] with length 0: panic

That error is recovered server-side and returned to the client as a generic RPC failure—without the server stack trace in test output. So the flaky test is not merely choosing between two benign branches; it can also mask a real bug in the trace timeout/race path (an out-of-bounds access somewhere in tracer result handling).

Goal

Replace timing-dependent integration assertions with deterministic unit-level coverage for timeout handling around TraceEthTxMsg, without relying on real scheduler races or localnet block timing. Integration tests should remain as smoke/compatibility checks for debug trace endpoints (debug_traceBlockByNumber, related block/tx trace RPCs), but must not encode strict expectations on "1ns" behavior.

Deterministic tests should validate, across repeated runs:

• Timeout triggers and tracer stop behavior when the deadline fires before trace completion.
• Successful completion when execution finishes before the timeout goroutine stops the tracer.
• Error propagation semantics (gRPC code, message shape) for timeout cancellation versus other failures.
• Once the root cause of the recovered [0] panic is identified, a regression test that reproduces it without nanosecond timing.

Acceptance criteria

• Unit tests pass reliably with -count=100 (or equivalent repeat runs) on timeout scenarios.
• Integration suite no longer fails nondeterministically due to the "1ns" case alone.
• No behavioral regression in existing debug trace RPC endpoints for normal timeout values.

Open questions

• What is the canonical client-visible error for trace timeout cancellation—is "execution timeout" with DeadlineExceeded the intended contract for JSON-RPC callers?
• Does the [0] panic originate in geth's callTracer, Nibiru's recover wrapper, or result serialization after tracer.GetResult()?

Related context

Comment in tracing_test.go (lines ~120–124) references this issue. Prior debugging command:

go test ./eth/rpc/rpcapi -run '^Test$' -testify.m '^TestTraceBlock$' -count=1 -v

[Unique-Divine]

Additional signal from tightening the current integration test:

While changing eth/rpc/rpcapi/tracing_test.go so the Timeout = "1ns" case only accepts either:

• err == nil (trace completed before the timeout goroutine stopped the tracer), or
• an error containing "execution timeout",

I reran the focused suite method:

go test ./eth/rpc/rpcapi -run '^Test$' -testify.m '^TestTraceBlock$' -count=1 -v

The stricter assertion exposed a third outcome:

Error "rpc error: code = Unknown desc = runtime error: index out of range [0] with length 0: panic" does not contain "execution timeout"

So the 1ns timeout path is not only timing-sensitive between success and timeout. It can also surface a recovered server-side panic. That suggests there is an actual bug in the trace timeout/race path that should be investigated separately from simply making the test less flaky.

Useful details:

• The panic happened during TestTraceBlock's ultra-small timeout case.
• The client-side test output did not include the server-side stack trace because the panic was recovered and returned as an RPC error.
• Next debugging step should be to capture the localnet/query server panic stack for this run and identify the exact [0] access.
• This strengthens the need for deterministic unit coverage around TraceEthTxMsg timeout behavior, plus a regression test for the recovered panic once the root cause is known.