# NousResearch/hermes-agent · Gateway process lifecycle and stale PID handling

## Open now

- 2 issues, 3 PRs open on GitHub.
- #53861: macOS: gateway stays down after update — launchd defers respawn in on-demand-only mode, update flow doesn't verify restart
- #56524: Telegram in-band hermes update can leave launchd gateway offline after drain timeout
- PR #21036: fix(gateway): exclude restart requester from drain
- PR #42338: fix(gateway): use canonical _pid_exists in restart watcher; /restart …
- PR #56908: fix(gateway): wait for launchd restart readiness in-band (#56524)

These members address bugs and fixes related to gateway startup, restart, and shutdown race conditions, particularly around stale or recycled PID detection and recovery from unexpected termination signals.

## TL;DR

- 39 member(s): 2 open issue(s), 15 closed issue(s), 3 open PR(s), 3 merged PR(s), 16 PR(s) closed without merging.
- A fix was merged to main in PR #28341. Verify it addresses your case.
- **Recommended:** read PR #28341 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-28341,pr-39000,pr-51734`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text.
- Members outside that structural core may belong to a different underlying problem.
- Pairwise link density `0.04`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Open related issues

- #53861: macOS: gateway stays down after update — launchd defers respawn in on-demand-only mode, update flow doesn't verify restart
- #56524: Telegram in-band hermes update can leave launchd gateway offline after drain timeout

## Closed / duplicate issues

- #5126: \`hermes gateway restart\` hangs on macOS due to KeepAlive race condition
- #9659: launchd plist KeepAlive should be unconditional, not SuccessfulExit-based
- #11258: Gateway self-restart/update can exit cleanly into draining state and stay dead under systemd Restart=on-failure
- #14176: \[Bug\]: Gateway hang on clean exit / restart race with stale PID
- #20694: Gateway self-restart from active WhatsApp chat can self-block graceful drain and always hit drain timeout
- #26198: hermes gateway restart on macOS (launchd) can leave gateway in stopped state
- #27592: Telegram gateway can stay down after self-restart if launchd job is unloaded
- #28135: hermes update via gateway leaves launchd service unrestarted on macOS
- #29180: \[Bug\]: /restart does not relaunch the gateway under macOS launchd
- #31048: hermes update fails to restart gateway on Linux systemd (exit code 75 handoff broken)
- #31162: \[BUG\] Gateway not auto-restarted after \`hermes update\`
- #37388: \[Bug\]: gateway launchd KeepAlive (SuccessfulExit:false) doesn't restart after a clean --replace exit on macOS
- #43475: /restart bricks a launchd-managed gateway on macOS — exits 0, KeepAlive.SuccessfulExit=false won't revive it
- #48746: issue
- #62128: \[Bug\]: macOS launchd: post-update gateway restart left service dead (respawn pended) and desktop app never relaunched itself

## Prior & competing PRs

- **#5127: fix: use launchctl kickstart for atomic gateway restart on macOS**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `74ff62f5ac803029b498dfb22b7322a9d3267a4b`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#9689: fix(gateway): use unconditional KeepAlive in launchd plist**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `c9b62061d43feb61a3a69caec99148623d988a5e`, `#37388`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#28341: fix(gateway): exit code 75 on service restart so launchd relaunches (#28200)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/run.py`
  - fix hunks:
  - `gateway/run.py`

    ```diff
    @@ -17346,6 +17346,19 @@ def restart_signal_handler():
             )
             return False  # → sys.exit(1) in the caller
     
    +    # When the gateway is restarting via the service manager (SIGUSR1 →
    +    # launchd_restart or /restart / /update commands), exit with code 75 so
    +    # that launchd's ``KeepAlive → SuccessfulExit → false`` policy treats
    +    # the exit as *unsuccessful* and relaunches the service.  This mirrors
    +    # the systemd ``RestartForceExitStatus=75`` convention already used by
    +    # the systemd unit template.
    +    if runner._restart_via_service:
    +        logger.info(
    +            "Exiting with code 75 (service-restart requested) so "
    +            "launchd KeepAlive relaunches the gateway."
    +        )
    +        raise SystemExit(75)
    +
         return True
     
     
    ```
- **#29181: fix(gateway): detect macOS launchd in service-restart path**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `abc3662bf6076045e4d4dc1e14a74cb35d69b86e`, `#43475`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#37411: fix(gateway): use unconditional KeepAlive in launchd plist so gateway restarts after clean --replace exit (#37388)**. closed without merge · fix tier `suggested`
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#39000`, `#38796`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#38796: fix(cli): launchd KeepAlive unconditional restart (#37388)**. closed without merge · fix tier `suggested`
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `c9b62061d`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#39000: fix(cli): launchd KeepAlive unconditional restart (#37388)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `hermes_cli/gateway.py`, `scripts/release.py`, `tests/hermes_cli/test_gateway_service.py`
  - fix hunks:
  - `hermes_cli/gateway.py`

    ```diff
    @@ -227,9 +227,8 @@ def _graceful_restart_via_sigusr1(pid: int, drain_timeout: float) -> bool:
     
         SIGUSR1 is wired in gateway/run.py to ``request_restart(via_service=True)``
         which drains in-flight agent runs (up to ``agent.restart_drain_timeout``
    -    seconds), then exits.  systemd relaunches clean exits via
    -    ``Restart=always``; launchd still uses a non-zero planned-restart exit
    -    because its plist has ``KeepAlive.SuccessfulExit = false``.
    +    seconds), then exits.  Both systemd (``Restart=always``) and launchd
    +    (unconditional ``<key>KeepAlive</key><true/>``) restart on any exit.
     
         This is the drain-aware alternative to ``systemctl restart`` / ``SIGTERM``,
         which SIGKILL in-flight agents after a short timeout.
    ```

    ```diff
    @@ -3083,10 +3082,7 @@ def generate_launchd_plist() -> str:
         <true/>
         
         <key>KeepAlive</key>
    -    <dict>
    -        <key>SuccessfulExit</key>
    -        <false/>
    -    </dict>
    +    <true/>
         
         <key>StandardOutPath</key>
         <string>{log_dir}/gateway.log</string>
    ```

    ```diff
    @@ -3249,7 +3245,7 @@ def launchd_stop():
             pass
         # bootout unloads the service definition so KeepAlive doesn't respawn
         # the process.  A plain `kill SIGTERM` only signals the process — launchd
    -    # immediately restarts it because KeepAlive.SuccessfulExit = false.
    +    # immediately restarts it because KeepAlive is unconditionally true.
         # `hermes gateway start` re-bootstraps when it detects the job is unloaded.
         try:
             subprocess.run(["launchctl", "bootout", target], check=True, timeout=90)
    ```
  - `scripts/release.py`

    ```diff
    @@ -72,6 +72,7 @@
         "[redacted:email]": "kchuang1015",
         "[redacted:email]": "MaheshtheDev",
         "[redacted:email]": "kyssta-exe",
    +    "[redacted:email]": "ashishpatel26",
         "[redacted:email]": "fujinice",
         "[redacted:email]": "carltonawong",
         "[redacted:email]": "EvilHumphrey",
    ```
  - `tests/hermes_cli/test_gateway_service.py`

    ```diff
    @@ -2579,3 +2579,24 @@ def test_launchd_workingdirectory_is_hermes_home(self, tmp_path, monkeypatch):
             assert m, "plist has no WorkingDirectory entry"
             assert Path(m.group(1)).resolve() == home.resolve()
             assert "/.worktrees/" not in m.group(1)
    +
    +    def test_launchd_plist_keepalive_unconditional(self, tmp_path, monkeypatch):
    +        """KeepAlive must be unconditional <true/> so the gateway restarts on clean exits.
    +
    +        Bug #37388: the old ``KeepAlive.SuccessfulExit = false`` dict form meant
    +        launchd would NOT restart after a zero-exit (e.g. ``gateway run --replace``
    +        causes the old instance to exit cleanly).  Switching to the scalar
    +        ``<key>KeepAlive</key><true/>`` makes launchd restart regardless of exit code.
    +        """
    +        home = tmp_path / ".hermes"
    +        home.mkdir()
    +        monkeypatch.setattr(gateway_cli, "get_hermes_home", lambda: home)
    +        plist = gateway_cli.generate_launchd_plist()
    +
    +        # Scalar <true/> must be present immediately after the KeepAlive key
    +        assert "<key>KeepAlive</key>" in plist
    +        # The unconditional form
    +        assert "<key>KeepAlive</key>\n    <true/>" in plist
    +        # The old conditional dict form must NOT appear
    +        assert "SuccessfulExit" not in plist
    +        assert "<key>KeepAlive</key>\n    <dict>" not in plist
    ```
- **#51112: fix(gateway): detect launchd via XPC_SERVICE_NAME for Telegram /restart path**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `abc3662bf6076045e4d4dc1e14a74cb35d69b86e`, `#43475`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#51734: fix(gateway): detect launchd in /restart service-manager probe (#43475)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/slash_commands.py`, `scripts/release.py`, `tests/gateway/test_restart_service_detection.py`
  - fix hunks:
  - `gateway/slash_commands.py`

    ```diff
    @@ -960,7 +960,15 @@ async def _handle_restart_command(self, event: MessageEvent) -> Union[str, Ephem
             # us.  The detached subprocess approach (setsid + bash) doesn't work
             # under systemd (KillMode=mixed kills the cgroup) or Docker (tini
             # exits when the gateway dies, taking the detached helper with it).
    -        _under_service = bool(os.environ.get("INVOCATION_ID"))  # systemd sets this
    +        # systemd sets INVOCATION_ID; launchd sets XPC_SERVICE_NAME to the
    +        # job label.  Without the launchd check, macOS /restart takes the
    +        # detached path and exits 0, which KeepAlive.SuccessfulExit=false
    +        # treats as a deliberate stop — the gateway stays dead until next
    +        # login.  Interactive macOS shells inherit XPC_SERVICE_NAME=0, so
    +        # "0" must count as not-under-launchd.
    +        _under_service = bool(os.environ.get("INVOCATION_ID")) or os.environ.get(
    +            "XPC_SERVICE_NAME", "0"
    +        ) not in ("", "0")
             _in_container = os.path.exists("/.dockerenv") or os.path.exists("/run/.containerenv")
             if _under_service or _in_container:
                 self.request_restart(detached=False, via_service=True)
    ```
  - `scripts/release.py`

    ```diff
    @@ -1547,6 +1547,7 @@
         "[redacted:email]": "bird",  # PR #25219 (gateway docker exit-75 restart)
         "[redacted:email]": "davidcampbelldc",  # PR #26834 (web_server proxy_headers=False)
         "[redacted:email]": "davidcampbelldc",
    +    "[redacted:email]": "chazmaniandinkle",  # PR #43888 (launchd /restart detection)
         "[redacted:email]": "hehehe0803",  # PR #26212 salvage (codex kanban writable root)
         "[redacted:email]": "hehehe0803",
         "[redacted:email]": "kasunvinod",  # PR #24126 salvage (codex timeout propagation)
    ```
  - `tests/gateway/test_restart_service_detection.py`

    ```diff
    @@ -0,0 +1,85 @@
    +"""Tests for /restart service-manager detection (launchd vs interactive).
    +
    +The /restart handler routes through ``request_restart(via_service=True)``
    +when a service manager supervises the gateway, so the process exits with
    +the service-restart code and the manager relaunches it.  Under macOS
    +launchd the plist uses ``KeepAlive.SuccessfulExit=false`` — a clean exit 0
    +is treated as a deliberate stop and the gateway stays dead (#43475) — so
    +launchd must be detected here in the handler, not only at the exit-code
    +site (which never runs unless ``via_service=True`` is already set).
    +
    +launchd sets ``XPC_SERVICE_NAME`` to the job label for processes it
    +spawns.  Interactive macOS shells inherit ``XPC_SERVICE_NAME=0`` (a
    +truthy string), so the probe must treat ``"0"`` as not-under-launchd:
    +routing an unsupervised interactive gateway to the service path would
    +make it exit non-zero with nothing to revive it.
    +"""
    +from unittest.mock import MagicMock
    +
    +import pytest
    +
    +import gateway.run as gateway_run
    +from gateway.platforms.base import MessageEvent, MessageType
    +from tests.gateway.restart_test_helpers import make_restart_runner, make_restart_source
    +
    +
    +def _make_restart_event(update_id: int | None = 100) -> MessageEvent:
    +    return MessageEvent(
    +        text="/restart",
    +        message_type=MessageType.TEXT,
    +        source=make_restart_source(),
    +        message_id="m1",
    +        platform_update_id=update_id,
    +    )
    +
    +
    +def _make_runner_with_mock_restart(tmp_path, monkeypatch):
    +    monkeypatch.setattr(gateway_run, "_hermes_home", tmp_path)
    +    monkeypatch.delenv("INVOCATION_ID", raising=False)
    +    monkeypatch.delenv("XPC_SERVICE_NAME", raising=False)
    +    runner, _adapter = make_restart_runner()
    +    runner.request_restart = MagicMock(return_value=True)
    +    return runner
    +
    +
    [redacted:email]
    +async def test_restart_under_launchd_uses_service_path(tmp_path, monkeypatch):
    +    """launchd job label in XPC_SERVICE_NAME routes /restart via the service path."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +    monkeypatch.setenv("XPC_SERVICE_NAME", "ai.hermes.gateway")
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=False, via_service=True)
    +
    +
    [redacted:email]
    +async def test_restart_in_interactive_macos_shell_uses_detached_path(tmp_path, monkeypatch):
    +    """XPC_SERVICE_NAME=0 (inherited by interactive macOS shells) is NOT a service."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +    monkeypatch.setenv("XPC_SERVICE_NAME", "0")
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=True, via_service=False)
    +
    +
    [redacted:email]
    +async def test_restart_without_service_env_uses_detached_path(tmp_path, monkeypatch):
    +    """No service-manager env at all falls back to the detached restart."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=True, via_service=False)
    +
    +
    [redacted:email]
    +async def test_restart_under_systemd_uses_service_path(tmp_path, monkeypatch):
    +    """INVOCATION_ID (systemd) still routes via the service path."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +    monkeypatch.setenv("INVOCATION_ID", "abc123")
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=False, via_service=True)
    ```
- **#10637: fix(gateway): use unconditional KeepAlive in launchd plist**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#9689`
- **#14332: fix(gateway): treat recycled PID with unreadable start_time as stale (#14176)**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `ccf526964a793371d4df97b4d1ae4f7d6fd63439`, `d682f320b35a13084371a541a835e1d988c982b8`, `#14176`, `#14128`
- **#21036: fix(gateway): exclude restart requester from drain**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `619bd782738adfa87544d0fda7a1114defe86e4c`
  - files: `gateway/run.py`, `tests/e2e/test_platform_commands.py`, `tests/gateway/restart_test_helpers.py`, `tests/gateway/test_gateway_shutdown.py`, `tests/gateway/test_restart_drain.py`, `tests/gateway/test_restart_notification.py`
- **#28200: fix(gateway): exit code 75 on service restart so launchd relaunches**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `5987b24314b3f40043ba150819ff698bce220db0`, `#28341`
- **#34464: fix(gateway): refresh systemd units before update restart**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`
- **#37534: fix(launchd): use unconditional KeepAlive so gateway restarts after clean exit #37388**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#39000`, `#38796`
- **#42338: fix(gateway): use canonical _pid_exists in restart watcher; /restart …**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`
  - files: `gateway/run.py`, `tests/gateway/test_restart_drain.py`
- **#43498: fix(gateway): exit non-zero on /restart so launchd revives the gateway**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#51734`, `#43475`
- **#43596: fix(gateway): exit non-zero on macOS launchd restart path (#43475)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#51734`, `#43475`
- **#43888: fix(gateway): detect launchd in the /restart service-manager probe**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `abc3662bf6076045e4d4dc1e14a74cb35d69b86e`, `#43475`, `#43475`, `#51734`
- **#53869: fix(cli): verify gateway liveness after macOS launchd restart (#53861)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `1bf93660f1`, `bee489d563`, `#88848`, `#74973`, `#53861`, `#99685`
- **#56908: fix(gateway): wait for launchd restart readiness in-band (#56524)**. open
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `#56524`, `#42450`, `#43181`, `#10317`
  - files: `hermes_cli/gateway.py`, `tests/hermes_cli/test_gateway_service.py`
- **#72752: fix(gateway): verify launchd service survivability after restart**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `1bf93660f1d`, `ac2e4207c11`

## Fix status

- Merged to main in PR #28341 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-06-24`.
- Subsystem: `apps/desktop`