Desktop app backend startup failures and timeouts

Fixes on record

Merged to main. A stamp counts the issues a fix is linked to.

• #50345 fix(desktop): close the cold-start boot loop on Windows (#50209)

  merged to main

  files: apps/desktop/electron/backend-ready.cjs, apps/desktop/electron/backend-ready.test.cjs, apps/desktop/package.json, hermes_cli/web_server.py, tests/hermes_cli/test_web_server_boot_handshake.py

  Show the diff

  • apps/desktop/electron/backend-ready.cjs

    @@ -1,5 +1,32 @@
     const _READY_RE = /^HERMES_DASHBOARD_READY port=(\d+)/m
     
    +// The announcement clock starts the instant the backend process is spawned —
    +// before uvicorn binds its socket. On a cold install the child must first
    +// compile and import the whole `hermes_cli.main` → `web_server` → FastAPI/
    +// uvicorn chain, and on Windows real-time AV (Defender) scans every freshly
    +// written `.pyc`. That pre-bind cost can run 30-60s on a slow disk, so a tight
    +// 45s deadline kills a *healthy but still-starting* backend and respawns it,
    +// piling up orphaned processes (issue #50209). A roomier default absorbs the
    +// cold-start cost; a warm start still announces in well under a second.
    +const DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS = 90_000
    +// Never trust a deadline tighter than the warm-start path needs; floor at 45s
    +// (the historical default) so a malformed override can't reintroduce the loop.
    +const MIN_PORT_ANNOUNCE_TIMEOUT_MS = 45_000
    +
    +/**
    + * Resolve the port-announcement deadline. Honors the
    + * HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS env override (for users on slow
    + * disks / aggressive AV who need an even longer cold-start window), clamped
    + * to a sane floor so a bad value can't make boot flakier than the default.
    + */
    +function resolvePortAnnounceTimeoutMs(env = process.env) {
    +  const parsed = Number(env.HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS)
    +  if (Number.isFinite(parsed) && parsed > 0) {
    +    return Math.max(MIN_PORT_ANNOUNCE_TIMEOUT_MS, Math.round(parsed))
    +  }
    +  return DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS
    +}
    +
     /**
      * Watch a child process's stdout for the `HERMES_DASHBOARD_READY port=<N>`
      * line that web_server.py prints after uvicorn binds its socket.
    @@ -9,11 +36,15 @@ const _READY_RE = /^HERMES_DASHBOARD_READY port=(\d+)/m
      *   - the child emits an `error` event
      *   - no line arrives within the timeout
      *
    + * The default timeout is cold-start tolerant (see
    + * DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS) because the clock starts before the
    + * backend has even bound its port. Pass an explicit `timeoutMs` to override.
    + *
      * A single `cleanup()` tears down every listener (data/exit/error/timeout)
      * on every terminal path — resolve, reject, or timeout — so repeated
      * backend spawns don't leak listener slots on the child.
      */
    -function waitForDashboardPort(child, timeoutMs = 45_000) {
    +function waitForDashboardPort(child, timeoutMs = resolvePortAnnounceTimeoutMs()) {
       return new Promise((resolve, reject) => {
         let buf = ''
         let done = false
    @@ -63,4 +94,9 @@ function waitForDashboardPort(child, timeoutMs = 45_000) {
       })
     }
     
    -module.exports = { waitForDashboardPort }
    +module.exports = {
    +  waitForDashboardPort,
    +  resolvePortAnnounceTimeoutMs,
    +  DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS,
    +  MIN_PORT_ANNOUNCE_TIMEOUT_MS,
    +}

  • apps/desktop/electron/backend-ready.test.cjs

    @@ -0,0 +1,121 @@
    +/**
    + * Tests for electron/backend-ready.cjs.
    + *
    + * Run with: node --test electron/backend-ready.test.cjs
    + * (Wired into npm test:desktop:platforms in package.json.)
    + *
    + * Covers the cold-start port-announcement deadline (issue #50209): the clock
    + * starts before the backend binds its port, so a tight 45s deadline killed a
    + * healthy-but-still-compiling backend on cold Windows installs. The default is
    + * now cold-start tolerant and overridable via
    + * HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS, clamped to a 45s floor.
    + */
    +
    +const test = require('node:test')
    +const assert = require('node:assert/strict')
    +const { EventEmitter } = require('node:events')
    +
    +const {
    +  waitForDashboardPort,
    +  resolvePortAnnounceTimeoutMs,
    +  DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS,
    +  MIN_PORT_ANNOUNCE_TIMEOUT_MS,
    +} = require('./backend-ready.cjs')
    +
    +// A minimal stand-in for a spawned child process: an EventEmitter with a
    +// stdout EventEmitter, matching the surface waitForDashboardPort consumes
    +// (child.stdout.on('data'), child.on('exit'|'error') + the .off() teardown).
    +function makeFakeChild() {
    +  const child = new EventEmitter()
    +  child.stdout = new EventEmitter()
    +  return child
    +}
    +
    +// ---------------------------------------------------------------------------
    +// resolvePortAnnounceTimeoutMs
    +// ---------------------------------------------------------------------------
    +
    +test('default is cold-start tolerant (> the historical 45s floor)', () => {
    +  assert.equal(resolvePortAnnounceTimeoutMs({}), DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS)
    +  assert.ok(
    +    DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS > MIN_PORT_ANNOUNCE_TIMEOUT_MS,
    +    'cold-start default must exceed the warm-start floor'
    +  )
    +})
    +
    +test('honors a valid HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS override', () => {
    +  const env = { HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS: '120000' }
    +  assert.equal(resolvePortAnnounceTimeoutMs(env), 120_000)
    +})
    +
    +test('clamps an override below the floor up to the 45s minimum', () => {
    +  const env = { HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS: '1000' }
    +  assert.equal(resolvePortAnnounceTimeoutMs(env), MIN_PORT_ANNOUNCE_TIMEOUT_MS)
    +})
    +
    +test('rounds a fractional override', () => {
    +  const env = { HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS: '60000.7' }
    +  assert.equal(resolvePortAnnounceTimeoutMs(env), 60_001)
    +})
    +
    +test('falls back to the default for malformed / non-positive overrides', () => {
    +  for (const bad of ['', 'abc', '0', '-5', 'NaN', undefined]) {
    +    const env = bad === undefined ? {} : { HERMES_DESKTOP_PORT_ANNOUNCE_TIMEOUT_MS: bad }
    +    assert.equal(
    +      resolvePortAnnounceTimeoutMs(env),
    +      DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS,
    +      `override ${JSON.stringify(bad)} should fall through to the default`
    +    )
    +  }
    +})
    +
    +// ---------------------------------------------------------------------------
    +// waitForDashboardPort
    +// ---------------------------------------------------------------------------
    +
    +test('resolves with the announced port', async () => {
    +  const child = makeFakeChild()
    +  const p = waitForDashboardPort(child, 1000)
    +  child.stdout.emit('data', 'noise before\nHERMES_DASHBOARD_READY port=54321\n')
    +  assert.equal(await p, 54321)
    +})
    +
    +test('parses the port even when the line arrives split across chunks', async () => {
    +  const child = makeFakeChild()
    +  const p = waitForDashboardPort(child, 1000)
    +  child.stdout.emit('data', 'HERMES_DASHBOARD_READY po')
    +  child.stdout.emit('data', 'rt=8080\n')
    +  assert.equal(await p, 8080)
    +})
    +
    +test('rejects when the child exits before announcing', async () => {
    +  const child = makeFakeChild()
    +  const p = waitForDashboardPort(child, 1000)
    +  child.emit('exit', 1, null)
    +  await assert.rejects(p, /exited before port announcement/)
    +})
    +
    +test('rejects on a child error event', async () => {
    +  const child = makeFakeChild()
    +  const p = waitForDashboardPort(child, 1000)
    +  child.emit('error', new Error('spawn ENOENT'))
    +  await assert.rejects(p, /spawn ENOENT/)
    +})
    +
    +test('rejects with the timeout message after the deadline', async () => {
    +  const child = makeFakeChild()
    +  await assert.rejects(
    +    waitForDashboardPort(child, 20),
    +    /Timed out waiting for Hermes backend port announcement \(20ms\)/
    +  )
    +})
    +
    +test('a late announcement after timeout does not throw (listeners torn down)', async () => {
    +  const child = makeFakeChild()
    +  await assert.rejects(waitForDashboardPort(child, 20), /Timed out/)
    +  // The orphaned backend may still print its READY line later; the watcher
    +  // must have detached so this emit is a no-op rather than a double-settle.
    +  assert.doesNotThrow(() => {
    +    child.stdout.emit('data', 'HERMES_DASHBOARD_READY port=9999\n')
    +  })
    +})

  • apps/desktop/package.json

    @@ -37,7 +37,7 @@
         "test:desktop:nsis": "node scripts/test-desktop.mjs nsis",
         "test:desktop:existing": "node scripts/test-desktop.mjs existing",
         "test:desktop:fresh": "node scripts/test-desktop.mjs fresh",
    -    "test:desktop:platforms": "node --test electron/bootstrap-platform.test.cjs electron/hardening.test.cjs electron/backend-env.test.cjs electron/backend-probes.test.cjs electron/bootstrap-runner.test.cjs electron/connection-config.test.cjs electron/dashboard-token.test.cjs electron/gateway-ws-probe.test.cjs electron/oauth-net-request.test.cjs electron/desktop-uninstall.test.cjs electron/session-windows.test.cjs electron/link-title-window.test.cjs electron/workspace-cwd.test.cjs electron/fs-read-dir.test.cjs electron/git-root.test.cjs electron/windows-child-process.test.cjs electron/update-remote.test.cjs electron/update-rebuild.test.cjs electron/windows-user-env.test.cjs",
    +    "test:desktop:platforms": "node --test electron/bootstrap-platform.test.cjs electron/hardening.test.cjs electron/backend-env.test.cjs electron/backend-probes.test.cjs electron/backend-ready.test.cjs electron/bootstrap-runner.test.cjs electron/connection-config.test.cjs electron/dashboard-token.test.cjs electron/gateway-ws-probe.test.cjs electron/oauth-net-request.test.cjs electron/desktop-uninstall.test.cjs electron/session-windows.test.cjs electron/link-title-window.test.cjs electron/workspace-cwd.test.cjs electron/fs-read-dir.test.cjs electron/git-root.test.cjs electron/windows-child-process.test.cjs electron/update-remote.test.cjs electron/update-rebuild.test.cjs electron/windows-user-env.test.cjs",
         "typecheck": "tsc -p . --noEmit",
         "lint": "eslint src/ electron/",
         "lint:fix": "eslint src/ electron/ --fix",

  • hermes_cli/web_server.py

    @@ -144,6 +144,22 @@ def _start_desktop_cron_ticker(stop_event: "threading.Event", interval: int = 60
         provider.start(stop_event, interval=interval)
     
     
    +def _warm_gateway_module() -> None:
    +    try:
    +        import hermes_cli.gateway  # noqa: F401
    +    except Exception:
    +        pass
    +
    +
    +def _resolve_restart_drain_timeout() -> float:
    +    try:
    +        from hermes_cli.gateway import _get_restart_drain_timeout
    +        return _get_restart_drain_timeout()
    +    except ImportError:
    +        from gateway.restart import DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT
    +        return DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT
    +
    +
     @asynccontextmanager
     async def _lifespan(app: "FastAPI"):
         app.state.event_channels = {}  # dict[str, set]
    @@ -154,6 +170,14 @@ async def _lifespan(app: "FastAPI"):
         # event loop during lifespan startup — see _get_event_state's docstring.
         app.state.chat_argv_lock = asyncio.Lock()
     
    +    # Fire hermes_cli.gateway import into a background thread so the event
    +    # loop is not blocked and HERMES_DASHBOARD_READY fires without delay.
    +    # On a cold Windows install the module chain triggers .pyc compilation
    +    # and Defender real-time scans that can stall the event loop for 15-30s.
    +    # Running in an executor means the cost is paid in a worker thread while
    +    # the server socket is already open and accepting probes.
    +    asyncio.get_event_loop().run_in_executor(None, _warm_gateway_module)
    +
         # Desktop-spawned backends (HERMES_DESKTOP=1) fire cron jobs themselves,
         # since the app has no gateway running the scheduler. Server `hermes
         # dashboard` is unaffected — it relies on its own gateway.
    @@ -1855,19 +1879,15 @@ async def get_status(profile: Optional[str] = None):
                 gateway_state=gateway_state,
             )
             # Resolved drain timeout (seconds) so NAS can size its poll deadline
    -        # without out-of-band knowledge.  Reuse the single resolver
    -        # (HERMES_RESTART_DRAIN_TIMEOUT env → config agent.restart_drain_timeout
    -        # → default) rather than re-deriving the precedence chain here.
    -        try:
    -            from hermes_cli.gateway import _get_restart_drain_timeout
    -
    -            restart_drain_timeout = _get_restart_drain_timeout()
    -        except ImportError:
    -            # Resolver moved/renamed — fall back to the real default so the
    -            # field stays a numeric poll-deadline hint, never None.
    -            from gateway.restart import DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT
    -
    -            restart_drain_timeout = DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT
    +        # without out-of-band knowledge.  Offload to a thread: on a cold
    +        # Windows install the first import of hermes_cli.gateway blocks the
    +        # asyncio event loop for 15-30s (.pyc compilation + Defender scans),
    +        # exceeding the desktop handshake's 15s socket timeout.  After the
    +        # first call the module is in sys.modules and run_in_executor returns
    +        # in microseconds.
    +        restart_drain_timeout = await asyncio.get_running_loop().run_in_executor(
    +            None, _resolve_restart_drain_timeout
    +        )
     
             # Dashboard auth gate (Phase 7): surface whether the gate is engaged
             # and which providers are registered so ``hermes status`` and the

  • tests/hermes_cli/test_web_server_boot_handshake.py

    @@ -0,0 +1,188 @@
    +"""
    +Integration tests for the desktop boot handshake fix (PR #50231 / issue #50209).
    +
    +Simulates a slow hermes_cli.gateway import (15-30 s on a fresh Windows install
    +with Defender scanning every new .pyc) by patching the two helpers that touch
    +the blocking import and measuring event-loop freedom + response latency.
    +
    +Three scenarios are covered:
    +
    +1. _lifespan fire-and-forget: patched _warm_gateway_module sleeps N seconds in
    +   a thread; TestClient startup must complete in << N seconds (event loop not
    +   blocked, HERMES_DASHBOARD_READY would fire immediately).
    +
    +2. get_status run_in_executor: patched _resolve_restart_drain_timeout sleeps N
    +   seconds in a thread; a concurrent fast endpoint (/api/version) must respond
    +   during the wait, proving the event loop stayed free.
    +
    +3. No orphan accumulation: three concurrent /api/status requests all receive a
    +   200 response — no socket timeouts, no connection resets.
    +"""
    +
    +from __future__ import annotations
    +
    +import asyncio
    +import time
    +import threading
    +from unittest.mock import patch
    +
    +import pytest
    +
    +import hermes_cli.web_server as web_server_mod
    +
    +SLOW_SECONDS = 3  # represents the Defender worst-case (scaled down for CI speed)
    +
    +
    +# ---------------------------------------------------------------------------
    +# Helpers
    +# ---------------------------------------------------------------------------
    +
    +def _make_slow_warm(seconds: float):
    +    """Return a _warm_gateway_module replacement that sleeps in the caller thread."""
    +    def _slow():
    +        time.sleep(seconds)
    +    return _slow
    +
    +
    +def _make_slow_drain(seconds: float):
    +    """Return a _resolve_restart_drain_timeout replacement that sleeps in thread."""
    +    def _slow():
    +        time.sleep(seconds)
    +        return 180.0
    +    return _slow
    +
    +
    +# ---------------------------------------------------------------------------
    +# Test 1 — _lifespan fire-and-forget does not block the event loop
    +# ---------------------------------------------------------------------------
    +
    +def test_lifespan_warmup_is_nonblocking():
    +    """
    +    _warm_gateway_module runs in an executor (fire-and-forget).
    +    Even if it sleeps for SLOW_SECONDS, TestClient startup must complete
    +    in well under that time — proving the event loop was never blocked and
    +    HERMES_DASHBOARD_READY would have fired without delay.
    +    """
    +    from fastapi.testclient import TestClient
    +
    +    with patch.object(web_server_mod, "_warm_gateway_module", _make_slow_warm(SLOW_SECONDS)):
    +        t0 = time.perf_counter()
    +        with TestClient(web_server_mod.app, raise_server_exceptions=False) as _client:
    +            startup_ms = (time.perf_counter() - t0) * 1000
    +
    +    # Startup must complete in under half of SLOW_SECONDS (generous margin).
    +    # If the import were synchronous, startup would block for >= SLOW_SECONDS.
    +    threshold_ms = (SLOW_SECONDS * 1000) / 2
    +    assert startup_ms < threshold_ms, (
    +        f"_lifespan blocked the event loop: startup took {startup_ms:.0f} ms "
    +        f"but slow import is {SLOW_SECONDS * 1000:.0f} ms — "
    +        f"fire-and-forget is not working."
    +    )
    +
    +
    +# ---------------------------------------------------------------------------
    +# Test 2 — get_status run_in_executor keeps event loop free for other requests
    +# ---------------------------------------------------------------------------
    +
    +def test_get_status_does_not_block_event_loop():
    +    """
    +    /api/status calls _resolve_restart_drain_timeout via run_in_executor.
    +    While that slow call is running in a thread, a concurrent fast request
    +    (/api/version) must still get a response — proving the event loop stayed
    +    free during the import.
    +    """
    +    import httpx
    +    from anyio import from_thread, to_thread
    +
    +    results: dict[str, float] = {}
    +    errors: list[str] = []
    +
    +    async def _run():
    +        transport = httpx.ASGITransport(app=web_server_mod.app)
    +        async with httpx.AsyncClient(
    +            transport=transport, base_url="http://test"
    +        ) as client:
    +            # Fire both requests concurrently
    +            async with asyncio.TaskGroup() as tg:
    +                async def _status():
    +                    t = time.perf_counter()
    +                    r = await client.get("/api/status", timeout=SLOW_SECONDS + 5)
    +                    results["status_ms"] = (time.perf_counter() - t) * 1000
    +                    results["status_code"] = r.status_code
    +
    +                async def _version():
    +                    # Small delay so /api/status starts first
    +                    await asyncio.sleep(0.1)
    +                    t = time.perf_counter()
    +                    r = await client.get("/api/version", timeout=5)
    +                    results["version_ms"] = (time.perf_counter() - t) * 1000
    +                    results["version_code"] = r.status_code
    +
    +                tg.create_task(_status())
    +                tg.create_task(_version())
    +
    +    with patch.object(
    +        web_server_mod, "_resolve_restart_drain_timeout", _make_slow_drain(SLOW_SECONDS)
    +    ):
    +        asyncio.run(_run())
    +
    +    # /api/version must have responded well before /api/status finished
    +    assert "version_ms" in results, "Fast endpoint never responded"
    +    assert "status_ms" in results, "/api/status never responded"
    +
    +    version_ms = results["version_ms"]
    +    status_ms = results["status_ms"]
    +
    +    # /api/version should respond in < SLOW_SECONDS (event loop free)
    +    assert version_ms < SLOW_SECONDS * 1000, (
    +        f"/api/version took {version_ms:.0f} ms — event loop was blocked by "
    +        f"/api/status (which waited {status_ms:.0f} ms for the slow import)."
    +    )
    +
    +    # /api/status itself eventually returns 200
    +    assert results.get("status_code") == 200, (
    +        f"/api/status returned {results.get('status_code')} instead of 200"
    +    )
    +
    +
    +# ---------------------------------------------------------------------------
    +# Test 3 — no orphan accumulation: concurrent probes all receive 200
    +# ---------------------------------------------------------------------------
    +
    +def test_concurrent_status_probes_all_respond():
    +    """
    +    Three concurrent /api/status requests must all receive HTTP 200.
    +    If the event loop were blocked, later requests would pile up and
    +    the desktop shell would eventually reset the connection (WinError 10054).
    +    """
    +    import httpx
    +
    +    PROBES = 3
    +    responses: list[int] = []
    +
    +    async def _run():
    +        transport = httpx.ASGITransport(app=web_server_mod.app)
    +        async with httpx.AsyncClient(
    +            transport=transport, base_url="http://test"
    +        ) as client:
    +            tasks = [
    +                client.get("/api/status", timeout=SLOW_SECONDS + 5)
    +                for _ in range(PROBES)
    +            ]
    +            results = await asyncio.gather(*tasks, return_exceptions=True)
    +            for r in results:
    +                if isinstance(r, Exception):
    +                    responses.append(-1)
    +                else:
    +                    responses.append(r.status_code)
    +
    +    with patch.object(
    +        web_server_mod, "_resolve_restart_drain_timeout", _make_slow_drain(SLOW_SECONDS)
    +    ):
    +        asyncio.run(_run())
    +
    +    failed = [c for c in responses if c != 200]
    +    assert not failed, (
    +        f"{len(failed)}/{PROBES} probes failed (codes: {responses}). "
    +        f"This would cause WinError 10054 and orphan accumulation on desktop."
    +    )

• #62718 fix(desktop): validate WebSocket token before backend ready

  merged to main

  files: apps/desktop/electron/main.ts

• #71104 fix(desktop): boot readiness probes a lightweight /api/health

  merged to main

  files: apps/desktop/electron/backend-health.test.ts, apps/desktop/electron/backend-health.ts, apps/desktop/electron/main.ts, hermes_cli/dashboard_auth/public_paths.py, hermes_cli/web_server.py, tests/hermes_cli/test_dashboard_auth_middleware.py, tests/hermes_cli/test_dashboard_auth_status_endpoint.py

• #100442 fix(desktop): never lose HERMES_BACKEND_READY emitted before the port wait attaches

  merged to main

  files: apps/desktop/electron/backend-ready.test.ts, apps/desktop/electron/backend-ready.ts, apps/desktop/electron/main.ts

• #122876 fix(serve): announce both ready tokens so a stale packaged parser still boots

  merged to main

  files: apps/desktop/scripts/after-pack-toolchain.test.mjs, apps/desktop/scripts/after-pack.mjs, apps/desktop/scripts/after-pack.test.mjs, apps/desktop/scripts/backend-ready-artifact.mjs, apps/desktop/scripts/backend-ready-artifact.test.mjs, hermes_cli/web_server.py, tests/hermes_cli/test_serve_mcp_discovery_after_bind.py, tests/hermes_cli/test_serve_port_in_use.py

• #122965 fix(cli): pin utf-8 decoding on all text-mode subprocess readers (Windows gbk crash)

  merged to main

  files: hermes_cli/boot_bootstrap.py, hermes_cli/gitlock.py, hermes_cli/kanban_pr_acceptance.py, hermes_cli/local_runtime/bootstrap.py, hermes_cli/local_runtime/hardware.py, hermes_cli/macos_signing.py, hermes_cli/macos_tcc_anchor.py, hermes_cli/main_desktop.py, hermes_cli/plugin_catalog.py, hermes_cli/plugin_validate.py, hermes_cli/plugins_provenance.py, hermes_cli/plugins_updates.py, hermes_cli/source_build.py, hermes_cli/sqlite_runtime.py, hermes_cli/update_cmd_fleet_checkout.py, hermes_cli/update_lock.py, hermes_cli/version_info.py, hermes_cli/web_routers/local_models.py, tests/hermes_cli/test_subprocess_text_encoding.py