What we see
Once a dev relay promotes us to hot, the connection survives only ~60s before being torn down, then reconnects (which resets the ~15-min hot-promotion clock, so we churn and rarely stay hot long enough to do useful work). Observed repeatedly in the 2026-06-18 tx-format run against the play.dev relays:
13:34:40 peer-1 Connection reset by peer (os error 104) (after a tx exchange + 63 keepalives)
13:35:41 peer-2/3 Connection reset by peer
13:36:44 peer-5 keepalive client exited — tearing down connection
This is the "60s RST" that #15 was thought to have closed. The KeepAlive responder is fine — serve_keepalive answered 63 keepalives on peer-1 with correct cookie echoes before the teardown. The problem is on the client side, plus likely a relay-initiated RST.
Root cause (client-side 60s)
Our keepalive client (spawn_keepalive, net-core/src/peer/peer_task.rs) pings every 20s and then keepalive::keep_alive does runner.recv().await? to wait for the relay's reply. That recv happens in the StServer state (we sent MsgKeepAlive: StClient -> StServer), and:
// net-core/src/protocols/keepalive/mod.rs
pub const TIMEOUT_SERVER: Duration = Duration::from_secs(60);
fn timeout(StServer) => Some(TIMEOUT_SERVER) // 60s
So our client waits exactly 60s for the relay's keepalive response, then errors and tears the whole connection down (spawn_keepalive returns → duplex_task watchdog tears down).
This is the mirror of the bug #15 fixed for the responder: cardano-node doesn't run keepalive on cold/warm peers, so when the relay demotes us from hot back to warm and goes quiet on keepalive, our client hits the 60s StServer timeout and kills an otherwise-healthy warm connection — resetting the hot-promotion clock.
There may also be a relay-initiated RST
Most teardowns were Connection reset by peer (the relay RST'ing us), with leios_notify / txsubmission merely noticing the dead socket first — and several fired right after a successful TxSubmission exchange. So there are possibly two effects: (a) our 60s client timeout on warm demotion, and (b) the relay actively RST'ing us (its recycle policy, or something we send post-tx such as the TxSubmission ack accounting, newly reachable now that #17 lets us get that far).
Suggested fix
Mirror #15's cold/warm tolerance on the keepalive client: don't treat a keepalive timeout as fatal while warm (e.g. only run/await the client toward hot peers, or back off instead of tearing down). Recommend a RUST_LOG=...keepalive=debug,...txsubmission=debug run first to cleanly separate "our 60s timeout" from "relay RST" and capture the last message before each RST.
Context
Surfaced while validating #17 (txsubmission codec) — that fix is confirmed working (zero CBOR decode errors throughout), this is the next thing blocking sustained TxSubmission against the dev relays. Not a codec problem.
What we see
Once a dev relay promotes us to hot, the connection survives only ~60s before being torn down, then reconnects (which resets the ~15-min hot-promotion clock, so we churn and rarely stay hot long enough to do useful work). Observed repeatedly in the 2026-06-18 tx-format run against the play.dev relays:
This is the "60s RST" that #15 was thought to have closed. The KeepAlive responder is fine —
serve_keepaliveanswered 63 keepalives on peer-1 with correct cookie echoes before the teardown. The problem is on the client side, plus likely a relay-initiated RST.Root cause (client-side 60s)
Our keepalive client (
spawn_keepalive,net-core/src/peer/peer_task.rs) pings every 20s and thenkeepalive::keep_alivedoesrunner.recv().await?to wait for the relay's reply. That recv happens in theStServerstate (we sentMsgKeepAlive:StClient -> StServer), and:So our client waits exactly 60s for the relay's keepalive response, then errors and tears the whole connection down (
spawn_keepalivereturns →duplex_taskwatchdog tears down).This is the mirror of the bug #15 fixed for the responder: cardano-node doesn't run keepalive on cold/warm peers, so when the relay demotes us from hot back to warm and goes quiet on keepalive, our client hits the 60s
StServertimeout and kills an otherwise-healthy warm connection — resetting the hot-promotion clock.There may also be a relay-initiated RST
Most teardowns were
Connection reset by peer(the relay RST'ing us), with leios_notify / txsubmission merely noticing the dead socket first — and several fired right after a successful TxSubmission exchange. So there are possibly two effects: (a) our 60s client timeout on warm demotion, and (b) the relay actively RST'ing us (its recycle policy, or something we send post-tx such as the TxSubmission ack accounting, newly reachable now that #17 lets us get that far).Suggested fix
Mirror #15's cold/warm tolerance on the keepalive client: don't treat a keepalive timeout as fatal while warm (e.g. only run/await the client toward hot peers, or back off instead of tearing down). Recommend a
RUST_LOG=...keepalive=debug,...txsubmission=debugrun first to cleanly separate "our 60s timeout" from "relay RST" and capture the last message before each RST.Context
Surfaced while validating #17 (txsubmission codec) — that fix is confirmed working (zero CBOR decode errors throughout), this is the next thing blocking sustained TxSubmission against the dev relays. Not a codec problem.