HUD link: snappy internal reconnect + resilience, relaxed visual alert

Addresses a tester whose HUD link died after ~2 min and only a Motoeye
reboot recovered it, plus the goal that reconnect be snappy internally
without flashing the disconnect alert on a brief blip.

Detection / reconnect (internal, fast):
- Phone OkHttp pingInterval 15s -> 5s: a frozen/half-open HUD (frame sends
  still succeed into the TCP buffer) is now caught by a missed pong in ~5s,
  not 15s, so the dial loop rediscovers + reconnects promptly.
- HUD Ktor pingPeriod 15s -> 5s, timeout 30s -> 12s: frees the connection
  (peer/status) and unblocks the incoming loop within ~12s of the phone
  vanishing instead of 30s.
- connSeq guard: a stale, slowly-timing-out handler can no longer clobber
  the status/peer of a newer connection from a fast reconnect.

Resilience (self-recover without a reboot):
- Partial CPU wake lock while the link is up (+ WAKE_LOCK permission) so the
  Ktor server, UDP beacon and watchdog can't be frozen by a suspend window.
- Watchdog interval 30s -> 10s so a wedged listener restarts in ~20-30s; it
  now also revives the mDNS advertise if it was down, without a full restart.

Visual alert stays relaxed: the disconnect splash remains gated by HudApp's
existing 10s reconnect grace, decoupled from the now-faster internal
detection so a quick reconnect never flashes it.

Diagnostics: HudServer records watchdogRestarts / lastEndReason /
lastDisconnectMs and logs the end reason + a per-connection token so the
next "link died" report carries the cause.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: eried

app/src/main/java/com/eried/eucplanet/service/hud/HudServer.kt

@@ -130,7 +130,13 @@ class HudServer @Inject constructor(
     private val lifecycleLock = Mutex()
 
     private val http: OkHttpClient = OkHttpClient.Builder()
-        .pingInterval(15, TimeUnit.SECONDS)
+        // 5s, not 15s: a frozen/half-open HUD keeps the TCP socket open, so
+        // outbound frame sends succeed into the OS buffer and never error --
+        // only a missed pong reveals the dead peer. On the local hotspot link
+        // (sub-10ms RTT) a 5s ping detects a dead HUD in ~5s and triggers the
+        // dial-loop's rediscover+reconnect, instead of the rider staring at a
+        // frozen HUD for 15s+. Cheap on a LAN.
+        .pingInterval(5, TimeUnit.SECONDS)
         .connectTimeout(4, TimeUnit.SECONDS)
         .readTimeout(0, TimeUnit.MILLISECONDS)
         .retryOnConnectionFailure(true)

hud/src/main/AndroidManifest.xml

@@ -11,6 +11,10 @@
     <uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
     <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
     <uses-permission android:name="android.permission.CHANGE_WIFI_MULTICAST_STATE" />
+    <!-- Partial CPU wake lock so the Ktor server, UDP beacon and self-heal
+         watchdog keep running if the device tries to suspend. A tester saw
+         the link freeze after ~2 min and need a HUD reboot to recover. -->
+    <uses-permission android:name="android.permission.WAKE_LOCK" />
 
     <!-- ACCESS_WIFI_STATE auto-implies <uses-feature android.hardware.wifi
          required=true>, which blocks install on aftermarket HUDs whose

hud/src/main/java/com/eried/eucplanet/hud/HudActivity.kt

@@ -110,7 +110,11 @@ class HudActivity : ComponentActivity() {
         controller.onCommand = { server.sendCommand(it) }
         server.start()
 
-        // Forward connection events into the UI status banner.
+        // Forward connection events into the controller. The VISUAL disconnect
+        // alert is debounced separately in HudApp (DISCONNECT_RECONNECT_GRACE_MS)
+        // so a brief blip the link recovers from never flashes the splash --
+        // the internal detection (fast heartbeats) and the phone's reconnect
+        // loop stay snappy regardless of when the rider SEES the alert.
         lifecycleScope.launch {
             server.status.collect { controller.updateStatus(it) }
         }

hud/src/main/java/com/eried/eucplanet/hud/net/HudServer.kt

@@ -67,17 +67,20 @@ class HudServer(private val context: Context) {
         /** Server bind port. Matches [HudDiscovery.DEFAULT_PORT] so testers
          *  don't need to change anything in the phone settings. */
         private val PORT = HudDiscovery.DEFAULT_PORT
-        /** How often the self-heal watchdog ticks (ms). 30 s is small
-         *  enough to catch a dead listener within a single missed
-         *  reconnect window, large enough that it never noticeably
-         *  loads the device. */
-        private const val WATCHDOG_INTERVAL_MS: Long = 30_000L
+        /** How often the self-heal watchdog ticks (ms). 10 s so a wedged
+         *  listener or a stalled mDNS/beacon is caught and restarted within
+         *  ~20-30 s -- snappy enough that the rider rarely sees it, but not
+         *  so tight it churns. The CPU wake lock (see [wakeLock]) keeps this
+         *  loop running even if the device tries to suspend. */
+        private const val WATCHDOG_INTERVAL_MS: Long = 10_000L
         /** Consecutive failed localhost probes before forcing a server
-         *  restart. Two misses (= ~60 s) covers an Android Doze blip
-         *  without acting on a transient. */
+         *  restart. Two misses (~20-30 s) rides out a transient localhost
+         *  blip without acting on it. */
         private const val WATCHDOG_FAIL_THRESHOLD: Int = 2
         /** Per-attempt timeout for the watchdog's localhost TCP probe (ms). */
         private const val TCP_PROBE_TIMEOUT_MS: Int = 1_500
+        /** Wake-lock tag; shows up in `dumpsys power` for debugging. */
+        private const val WAKE_LOCK_TAG = "eucplanet-hud:server"
     }
 
     /** Connection state surfaced to the UI status banner. */
@@ -129,6 +132,23 @@ class HudServer(private val context: Context) {
     private var server: io.ktor.server.engine.ApplicationEngine? = null
     private var jmdns: JmDNS? = null
     private var multicastLock: WifiManager.MulticastLock? = null
+    /** CPU wake lock held while the link is up. The HUD keeps the screen on
+     *  (FLAG_KEEP_SCREEN_ON) which usually keeps the CPU running, but some
+     *  aftermarket HUDs power the panel independently or background the app;
+     *  a partial wake lock guarantees the Ktor server, beacon and watchdog
+     *  coroutines keep ticking so the link can't silently freeze the way a
+     *  tester saw it die after ~2 min. */
+    private var wakeLock: android.os.PowerManager.WakeLock? = null
+
+    // --- Diagnostics (surfaced on the HUD stats card + logcat) so the next
+    // "link died, had to reboot" report carries the cause, not just the
+    // symptom. ---
+    /** How many times the watchdog has force-restarted the server this run. */
+    @Volatile var watchdogRestarts: Int = 0; private set
+    /** Why the last phone connection ended (clean close / timeout / exception). */
+    @Volatile var lastEndReason: String = ""; private set
+    /** Wall-clock of the last disconnect, 0 if still on the first connection. */
+    @Volatile var lastDisconnectMs: Long = 0L; private set
     /** Self-heal watchdog. Polls the local listener and the current IPv4 every
      *  [WATCHDOG_INTERVAL_MS] so a hotspot IP-renew or a silently-dead Ktor
      *  socket gets fixed without the rider having to reboot the HUD. */
@@ -137,6 +157,10 @@ class HudServer(private val context: Context) {
      *  We require [WATCHDOG_FAIL_THRESHOLD] in a row before a full restart so
      *  a transient localhost blip doesn't churn the server. */
     @Volatile private var watchdogFailStreak: Int = 0
+    /** Monotonic connection counter so a stale, slowly-timing-out WebSocket
+     *  handler doesn't clobber the status/peer of a newer connection that
+     *  arrived during a fast reconnect. */
+    private val connSeq = java.util.concurrent.atomic.AtomicInteger(0)
     // Always-on UDP broadcast beacon so the phone can find us on networks
     // where mDNS multicast is blocked (most phone hotspots, every carrier
     // mobile hotspot). Runs in parallel with the mDNS advertise -- whichever
@@ -163,21 +187,31 @@ class HudServer(private val context: Context) {
 
     private suspend fun doStart() {
         if (server != null) return
+        acquireWakeLock()
         _localIp.value = pickLocalIp()
         val s = embeddedServer(CIO, port = PORT, host = "0.0.0.0") {
             install(WebSockets) {
-                // Heartbeat: phone or HUD network can go away silently. Ping
-                // every 15s so the OS surfaces a broken TCP connection
-                // within a reasonable window without spamming the wire.
+                // Heartbeat: phone or HUD network can go away silently. On the
+                // local hotspot link (sub-10ms RTT) we can afford an aggressive
+                // beat: ping every 5s and declare the peer dead after 12s of
+                // silence. This frees [_peer]/[_status] and unblocks the
+                // `incoming` loop within ~12s of the phone vanishing instead of
+                // 30s -- so a fresh dial from the phone is accepted promptly and
+                // the rider isn't stuck on a stale "connected" HUD.
                 // Millis form because the Duration property accessor names
                 // moved across Ktor 2.x; millis is stable.
-                pingPeriodMillis = 15_000L
-                timeoutMillis = 30_000L
+                pingPeriodMillis = 5_000L
+                timeoutMillis = 12_000L
             }
             routing {
                 webSocket(HudDiscovery.PATH_STATE) {
                     val remote = call.request.local.remoteHost
-                    Log.i(TAG, "phone connected: $remote")
+                    // Token for THIS connection. With fast heartbeats a phone
+                    // can redial while a previous handler is still unwinding its
+                    // timeout; we must not let the stale handler's finally clear
+                    // the status/peer that the newer connection already set.
+                    val myConn = connSeq.incrementAndGet()
+                    Log.i(TAG, "phone connected: $remote (#$myConn)")
                     _peer.value = remote
                     _status.value = Status.CONNECTED
                     // Diagnostic: remember the phone IP across reconnects
@@ -259,9 +293,17 @@ class HudServer(private val context: Context) {
                         endReason = "exception: ${t::class.simpleName} ${t.message}"
                     } finally {
                         sender.cancel()
-                        Log.i(TAG, "phone disconnected: $remote ($endReason)")
-                        _peer.value = null
-                        _status.value = Status.LISTENING
+                        Log.i(TAG, "phone disconnected: $remote (#$myConn, $endReason)")
+                        lastEndReason = endReason
+                        lastDisconnectMs = System.currentTimeMillis()
+                        // Only surrender the connection state if no newer
+                        // connection has taken over since we opened. Otherwise
+                        // a slow-timing-out stale handler would wrongly flip the
+                        // HUD back to LISTENING while a fresh phone is connected.
+                        if (connSeq.get() == myConn) {
+                            _peer.value = null
+                            _status.value = Status.LISTENING
+                        }
                     }
                 }
             }
@@ -303,6 +345,30 @@ class HudServer(private val context: Context) {
         server = null
         _peer.value = null
         _status.value = Status.LISTENING
+        releaseWakeLock()
+    }
+
+    /** Acquire a partial CPU wake lock so the link can't be frozen by a
+     *  device suspend window. Idempotent; safe to call on every (re)start. */
+    private fun acquireWakeLock() {
+        if (wakeLock?.isHeld == true) return
+        try {
+            val pm = context.applicationContext
+                .getSystemService(Context.POWER_SERVICE) as? android.os.PowerManager
+            wakeLock = pm?.newWakeLock(
+                android.os.PowerManager.PARTIAL_WAKE_LOCK, WAKE_LOCK_TAG
+            )?.apply {
+                setReferenceCounted(false)
+                acquire()
+            }
+        } catch (t: Throwable) {
+            Log.w(TAG, "wake lock acquire failed: ${t.message}")
+        }
+    }
+
+    private fun releaseWakeLock() {
+        try { if (wakeLock?.isHeld == true) wakeLock?.release() } catch (_: Throwable) {}
+        wakeLock = null
     }
 
     /** Watchdog loop. Two jobs per tick:
@@ -334,6 +400,12 @@ class HudServer(private val context: Context) {
                 try { multicastLock?.release() } catch (_: Throwable) {}
                 multicastLock = null
                 startMdnsAdvertise()
+            } else if (jmdns == null && !current.isNullOrBlank()) {
+                // mDNS advertise failed at startup (or got torn down) but the
+                // IP is fine -- revive it so phones browsing _eucplanet._tcp
+                // can still resolve us without a server restart.
+                Log.i(TAG, "watchdog: mDNS advertise was down; reviving")
+                startMdnsAdvertise()
             }
             // 2. Server-liveness probe.
             val alive = tcpProbe("127.0.0.1", PORT)
@@ -346,6 +418,7 @@ class HudServer(private val context: Context) {
                     Log.w(TAG, "watchdog: server appears wedged after $watchdogFailStreak " +
                         "consecutive misses; forcing restart")
                     watchdogFailStreak = 0
+                    watchdogRestarts++
                     // Restart in its own coroutine so the watchdog loop itself
                     // doesn't suspend on the start/stop lock that doStart()
                     // takes. If the watchdog held its own lifecycle-lock here