# NousResearch/hermes-agent · Gateway service restart loops and exit code fixes

## Open now

- 3 issues, 6 PRs open on GitHub.
- #24344: \[Bug\]: Systemd gateway service exits with status 1 on intentional stop
- #41631: \[Bug\]: gateway exits code 1 (→ unit 'failed') on systemctl stop; planned stops should exit 0
- #57739: \[Bug\]: macOS gateway enters restart loop — hardcoded --replace in launchd plist and incorrect exit code on SIGTERM
- PR #24351: fix(gateway): make intentional systemd stops exit cleanly
- PR #41642: fix(gateway): exit 0 on systemctl stop instead of exit 1 (failed unit)
- PR #41690: fix(gateway): exit 0 when systemd sends SIGTERM via systemctl stop
- PR #42555: fix(gateway): write planned-stop marker from systemd ExecStop
- PR #63535: fix(gateway): prevent macOS gateway restart loop on unexpected SIGTERM
- PR #85901: fix(gateway): mark direct systemd stops as planned

These issues and fixes address gateway daemon behavior under systemd, launchd, and other service managers, focusing on unintended restart loops caused by incorrect exit codes on intentional stops, the --replace flag in generated service definitions, and stale or colliding PID file handling.

## TL;DR

- 39 member(s): 3 open issue(s), 9 closed issue(s), 6 open PR(s), 4 merged PR(s), 17 PR(s) closed without merging.
- A fix was merged to main in PR #614. Verify it addresses your case.
- **Recommended:** read PR #614 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-614,pr-9875,pr-53903`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text.
- Members outside that structural core may belong to a different underlying problem.
- Pairwise link density `0.04`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Open related issues

- #24344: \[Bug\]: Systemd gateway service exits with status 1 on intentional stop
- #41631: \[Bug\]: gateway exits code 1 (→ unit 'failed') on systemctl stop; planned stops should exit 0
- #57739: \[Bug\]: macOS gateway enters restart loop — hardcoded --replace in launchd plist and incorrect exit code on SIGTERM

## Closed / duplicate issues

- #576: \[Bug\]: systemd restart loop (non-TTY CLI exit and "gateway already running" conflict)
- #5646: \[Bug\]: gateway restart / --replace flag causes gateway to exit immediately (status=0)
- #21549: fix(gateway): launchd double-spawn triggers infinite restart death spiral (exit code 1 on instance detection)
- #21915: \[systemd\] Incomplete process cleanup during restart causes port conflict and infinite restart loop
- #23272: Gateway restart loop: hardcoded --replace + systemd ExecStopPost causes infinite restart cycle
- #27275: Generated Linux gateway service can flap forever because ExecStart uses gateway run --replace under Type=simple
- #32951: Avoid systemd gateway restart/takeover loops from Restart=always + --replace
- #36213: \[Bug\] Windows: hermes gateway restart does not verify old process termination, causing silent Gateway failure
- #42517: fix(gateway): ExecStop should write planned-stop marker instead of inferring signal source

## Prior & competing PRs

- **#614: fix: resolve systemd restart loop with --replace flag**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/run.py`, `hermes_cli/gateway.py`, `hermes_cli/main.py`
- **#9875: fix: gateway auto-recovers from unexpected SIGTERM via systemd (#5646)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/run.py`, `gateway/status.py`
  - fix hunks:
  - `gateway/run.py`

    ```diff
    @@ -9261,8 +9261,18 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool =
     
         runner = GatewayRunner(config)
         
    +    # Track whether a signal initiated the shutdown (vs. internal request).
    +    # When an unexpected SIGTERM kills the gateway, we exit non-zero so
    +    # systemd's Restart=on-failure revives the process.  systemctl stop
    +    # is safe: systemd tracks stop-requested state independently of exit
    +    # code, so Restart= never fires for a deliberate stop.
    +    _signal_initiated_shutdown = False
    +
         # Set up signal handlers
         def shutdown_signal_handler():
    +        nonlocal _signal_initiated_shutdown
    +        _signal_initiated_shutdown = True
    +        logger.info("Received SIGTERM/SIGINT — initiating shutdown")
             asyncio.create_task(runner.stop())
     
         def restart_signal_handler():
    ```

    ```diff
    @@ -9332,6 +9342,21 @@ def restart_signal_handler():
         if runner.exit_code is not None:
             raise SystemExit(runner.exit_code)
     
    +    # When a signal (SIGTERM/SIGINT) caused the shutdown and it wasn't a
    +    # planned restart (/restart, /update, SIGUSR1), exit non-zero so
    +    # systemd's Restart=on-failure revives the process.  This covers:
    +    #   - hermes update killing the gateway mid-work
    +    #   - External kill commands
    +    #   - WSL2/container runtime sending unexpected signals
    +    # systemctl stop is safe: systemd tracks "stop requested" state
    +    # independently of exit code, so Restart= never fires for it.
    +    if _signal_initiated_shutdown and not runner._restart_requested:
    +        logger.info(
    +            "Exiting with code 1 (signal-initiated shutdown without restart "
    +            "request) so systemd Restart=on-failure can revive the gateway."
    +        )
    +        return False  # → sys.exit(1) in the caller
    +
         return True
     
     
    ```
  - `gateway/status.py`

    ```diff
    @@ -266,9 +266,25 @@ def read_runtime_status() -> Optional[dict[str, Any]]:
     
     
     def remove_pid_file() -> None:
    -    """Remove the gateway PID file if it exists."""
    +    """Remove the gateway PID file, but only if it belongs to this process.
    +
    +    During --replace handoffs, the old process's atexit handler can fire AFTER
    +    the new process has written its own PID file.  Blindly removing the file
    +    would delete the new process's record, leaving the gateway running with no
    +    PID file (invisible to ``get_running_pid()``).
    +    """
         try:
    -        _get_pid_path().unlink(missing_ok=True)
    +        path = _get_pid_path()
    +        record = _read_json_file(path)
    +        if record is not None:
    +            try:
    +                file_pid = int(record["pid"])
    +            except (KeyError, TypeError, ValueError):
    +                file_pid = None
    +            if file_pid is not None and file_pid != os.getpid():
    +                # PID file belongs to a different process — leave it alone.
    +                return
    +        path.unlink(missing_ok=True)
         except Exception:
             pass
     
    ```
- **#51734: fix(gateway): detect launchd in /restart service-manager probe (#43475)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/slash_commands.py`, `scripts/release.py`, `tests/gateway/test_restart_service_detection.py`
  - fix hunks:
  - `gateway/slash_commands.py`

    ```diff
    @@ -960,7 +960,15 @@ async def _handle_restart_command(self, event: MessageEvent) -> Union[str, Ephem
             # us.  The detached subprocess approach (setsid + bash) doesn't work
             # under systemd (KillMode=mixed kills the cgroup) or Docker (tini
             # exits when the gateway dies, taking the detached helper with it).
    -        _under_service = bool(os.environ.get("INVOCATION_ID"))  # systemd sets this
    +        # systemd sets INVOCATION_ID; launchd sets XPC_SERVICE_NAME to the
    +        # job label.  Without the launchd check, macOS /restart takes the
    +        # detached path and exits 0, which KeepAlive.SuccessfulExit=false
    +        # treats as a deliberate stop — the gateway stays dead until next
    +        # login.  Interactive macOS shells inherit XPC_SERVICE_NAME=0, so
    +        # "0" must count as not-under-launchd.
    +        _under_service = bool(os.environ.get("INVOCATION_ID")) or os.environ.get(
    +            "XPC_SERVICE_NAME", "0"
    +        ) not in ("", "0")
             _in_container = os.path.exists("/.dockerenv") or os.path.exists("/run/.containerenv")
             if _under_service or _in_container:
                 self.request_restart(detached=False, via_service=True)
    ```
  - `scripts/release.py`

    ```diff
    @@ -1547,6 +1547,7 @@
         "[redacted:email]": "bird",  # PR #25219 (gateway docker exit-75 restart)
         "[redacted:email]": "davidcampbelldc",  # PR #26834 (web_server proxy_headers=False)
         "[redacted:email]": "davidcampbelldc",
    +    "[redacted:email]": "chazmaniandinkle",  # PR #43888 (launchd /restart detection)
         "[redacted:email]": "hehehe0803",  # PR #26212 salvage (codex kanban writable root)
         "[redacted:email]": "hehehe0803",
         "[redacted:email]": "kasunvinod",  # PR #24126 salvage (codex timeout propagation)
    ```
  - `tests/gateway/test_restart_service_detection.py`

    ```diff
    @@ -0,0 +1,85 @@
    +"""Tests for /restart service-manager detection (launchd vs interactive).
    +
    +The /restart handler routes through ``request_restart(via_service=True)``
    +when a service manager supervises the gateway, so the process exits with
    +the service-restart code and the manager relaunches it.  Under macOS
    +launchd the plist uses ``KeepAlive.SuccessfulExit=false`` — a clean exit 0
    +is treated as a deliberate stop and the gateway stays dead (#43475) — so
    +launchd must be detected here in the handler, not only at the exit-code
    +site (which never runs unless ``via_service=True`` is already set).
    +
    +launchd sets ``XPC_SERVICE_NAME`` to the job label for processes it
    +spawns.  Interactive macOS shells inherit ``XPC_SERVICE_NAME=0`` (a
    +truthy string), so the probe must treat ``"0"`` as not-under-launchd:
    +routing an unsupervised interactive gateway to the service path would
    +make it exit non-zero with nothing to revive it.
    +"""
    +from unittest.mock import MagicMock
    +
    +import pytest
    +
    +import gateway.run as gateway_run
    +from gateway.platforms.base import MessageEvent, MessageType
    +from tests.gateway.restart_test_helpers import make_restart_runner, make_restart_source
    +
    +
    +def _make_restart_event(update_id: int | None = 100) -> MessageEvent:
    +    return MessageEvent(
    +        text="/restart",
    +        message_type=MessageType.TEXT,
    +        source=make_restart_source(),
    +        message_id="m1",
    +        platform_update_id=update_id,
    +    )
    +
    +
    +def _make_runner_with_mock_restart(tmp_path, monkeypatch):
    +    monkeypatch.setattr(gateway_run, "_hermes_home", tmp_path)
    +    monkeypatch.delenv("INVOCATION_ID", raising=False)
    +    monkeypatch.delenv("XPC_SERVICE_NAME", raising=False)
    +    runner, _adapter = make_restart_runner()
    +    runner.request_restart = MagicMock(return_value=True)
    +    return runner
    +
    +
    [redacted:email]
    +async def test_restart_under_launchd_uses_service_path(tmp_path, monkeypatch):
    +    """launchd job label in XPC_SERVICE_NAME routes /restart via the service path."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +    monkeypatch.setenv("XPC_SERVICE_NAME", "ai.hermes.gateway")
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=False, via_service=True)
    +
    +
    [redacted:email]
    +async def test_restart_in_interactive_macos_shell_uses_detached_path(tmp_path, monkeypatch):
    +    """XPC_SERVICE_NAME=0 (inherited by interactive macOS shells) is NOT a service."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +    monkeypatch.setenv("XPC_SERVICE_NAME", "0")
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=True, via_service=False)
    +
    +
    [redacted:email]
    +async def test_restart_without_service_env_uses_detached_path(tmp_path, monkeypatch):
    +    """No service-manager env at all falls back to the detached restart."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=True, via_service=False)
    +
    +
    [redacted:email]
    +async def test_restart_under_systemd_uses_service_path(tmp_path, monkeypatch):
    +    """INVOCATION_ID (systemd) still routes via the service path."""
    +    runner = _make_runner_with_mock_restart(tmp_path, monkeypatch)
    +    monkeypatch.setenv("INVOCATION_ID", "abc123")
    +
    +    await runner._handle_restart_command(_make_restart_event())
    +
    +    runner.request_restart.assert_called_once_with(detached=False, via_service=True)
    ```
- **#53903: docs(gateway): warn against custom ExecStopPost kill drop-in (restart loop)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `website/docs/user-guide/messaging/index.md`
- **#580: fix: make gateway run idempotent when already running**. closed without merge
  - maintainer direction: category `partial_fix`; implication `needs_llm`
- **#582: fix: prevent systemd restart loop from stale/colliding PID files**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#614`
- **#583: fix(gateway): make systemd startup idempotent and service-safe**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`
- **#613: fix(gateway): avoid systemd restart loop when instance already running**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#614`
- **#21555: fix(gateway): exit cleanly when another instance detected to avoid launchd restart loop**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `#51734`, `#37388`, `#43475`, `#21549`, `#42446`, `#42450`, `#28561`, `#21549`
- **#21907: fix(gateway): avoid --replace in generated services**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#23281`, `#23272`
- **#22391: fix(gateway): exit cleanly under systemd when duplicate instance detected**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`
- **#23281: fix(gateway): stop default replace in service runs**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `f556edc10`
- **#23330: fix(gateway): remove --replace from systemd/launchd service definitions (fixes #23272)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#26186`
- **#23350: fix(gateway): skip --replace when running under systemd**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#23330`, `#23281`, `#23272`
- **#24351: fix(gateway): make intentional systemd stops exit cleanly**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `002c45998`
  - files: `hermes_cli/gateway.py`, `hermes_cli/main.py`, `tests/hermes_cli/test_gateway.py`, `tests/hermes_cli/test_gateway_service.py`
- **#26186: fix(gateway): remove --replace flag from systemd/launchd/windows service definitions**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`
- **#32989: fix(gateway): avoid systemd restart loops on replace**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `4d18717b6c798d4f6bab9e736c6ed10c5a8365f4`, `#32951`
- **#36223: fix(gateway): wait for Windows restart slot before start**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `e7d2f0b93ca29b2cd95d6623fe8af7646173a75b`
- **#41639: fix(gateway): treat SIGTERM under systemd as planned stop (exit 0)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#41642`, `#41631`
- **#41642: fix(gateway): exit 0 on systemctl stop instead of exit 1 (failed unit)**. open
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#41639`, `#41631`
  - files: `gateway/run.py`, `tests/gateway/test_systemd_stop_exit_code.py`
- **#41690: fix(gateway): exit 0 when systemd sends SIGTERM via systemctl stop**. open
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#41642`, `#41639`, `#41631`
  - files: `gateway/run.py`, `tests/test_issue_41631_fix.py`
- **#42555: fix(gateway): write planned-stop marker from systemd ExecStop**. open
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#43236`, `#42675`, `#41631`, `#43236`, `#43236`
  - files: `hermes_cli/gateway.py`, `hermes_cli/subcommands/gateway.py`, `tests/hermes_cli/test_gateway_service.py`
- **#57743: fix(gateway): prevent macOS gateway restart loop on unexpected SIGTERM**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`
- **#57746: fix(gateway): prevent macOS gateway restart loop on unexpected SIGTERM**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`
- **#63535: fix(gateway): prevent macOS gateway restart loop on unexpected SIGTERM**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`
  - files: `gateway/restart.py`, `gateway/run.py`, `hermes_cli/gateway.py`, `tests/gateway/test_runner_startup_failures.py`, `tests/hermes_cli/test_gateway.py`, `tests/hermes_cli/test_gateway_service.py`
- **#85901: fix(gateway): mark direct systemd stops as planned**. open
  - maintainer direction: category `wontfix_bydesign`; implication `wontfix`; refs `#85901`
  - files: `gateway/run.py`, `gateway/run_shutdown.py`, `gateway/status.py`, `hermes_cli/gateway.py`, `hermes_systemd_planned_stop.py`, `tests/gateway/test_gateway_shutdown.py`, `tests/gateway/test_systemd_planned_stop.py`, `tests/gateway/test_systemd_stop_mark.py`, `tests/hermes_cli/test_systemd_planned_stop_unit.py`
- **#113358: fix(gateway): mark planned systemd stops**. closed without merge
  - maintainer direction: category `stale_abandoned`; implication `active`; refs `#24351`, `#42517`, `#42555`, `#85901`, `#24351`

## Fix status

- Merged to main in PR #614 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-06-28`.
- Subsystem: `gateway`