systemd timeout mismatch and restart loop fixes

Fixes on record

Merged to main. A stamp counts the issues a fix is linked to.

• #54066 fix(gateway): default restart_drain_timeout to 0 to kill systemd crash loop

  merged to main

  files: cli-config.yaml.example, gateway/run.py, hermes_cli/config.py, tests/gateway/test_gateway_shutdown.py

  Show the diff

  • cli-config.yaml.example

    @@ -626,10 +626,10 @@ agent:
       # gateway_timeout_warning: 900
     
       # Graceful drain timeout for gateway stop/restart (seconds).
    -  # The gateway stops accepting new work, waits for in-flight agents to
    -  # finish, then interrupts anything still running after this timeout.
    -  # 0 = no drain, interrupt immediately.
    -  # restart_drain_timeout: 60
    +  # Default 0 = no drain: a restart interrupts in-flight agents immediately,
    +  # cleans up, and exits. Set a positive value only if you want a grace
    +  # window on /restart, and keep it well under systemd's TimeoutStopSec.
    +  # restart_drain_timeout: 0
     
       # Max app-level retry attempts for API errors (connection drops, provider
       # timeouts, 5xx, etc.) before the agent surfaces the failure. Lower this

  • gateway/run.py

    @@ -5811,7 +5811,7 @@ async def start(self) -> bool:
                     logger.warning(
                         "Stale systemd unit detected: %s has TimeoutStopSec=%.0fs but "
                         "drain_timeout=%.0fs (expected >=%.0fs). systemd may SIGKILL the "
    -                    "gateway mid-drain. Run `hermes gateway service install --replace` "
    +                    "gateway mid-drain. Run `hermes gateway install --force` "
                         "to regenerate the unit, or shorten agent.restart_drain_timeout.",
                         _alignment.get("unit", "(unknown)"),
                         _alignment["timeout_stop_sec"],

  • hermes_cli/config.py

    @@ -916,13 +916,18 @@ def _ensure_hermes_home_managed(home: Path):
             # Graceful drain timeout for gateway stop/restart (seconds).
             # The gateway stops accepting new work, waits for running agents
             # to finish, then interrupts any remaining runs after the timeout.
    -        # 0 = no drain, interrupt immediately.
    +        # 0 = no drain, interrupt immediately (the default).
             #
    -        # 180s is calibrated for realistic in-flight agent turns: a typical
    -        # coding conversation mid-reasoning runs 60–150s per call, so a 60s
    -        # budget routinely interrupted legitimate work on /restart. Raise
    -        # further in config.yaml if you run very-long-reasoning models.
    -        "restart_drain_timeout": 180,
    +        # Contract: if you restart the gateway, in-flight work stops. We do
    +        # not hold the restart open for a grace window — a drain timeout
    +        # large enough to "save" a long agent turn would have to outlast an
    +        # unbounded task (some runs take days), which is impossible, and a
    +        # drain timeout shorter than systemd's TimeoutStopSec invites a
    +        # SIGKILL-mid-cleanup race that leaves a stale lock and crash-loops
    +        # the service. 0 sidesteps both: interrupt now, clean up, exit fast.
    +        # Set a positive value in config.yaml only if you explicitly want a
    +        # grace window on /restart (and keep it well under TimeoutStopSec).
    +        "restart_drain_timeout": 0,
             # Max app-level retry attempts for API errors (connection drops,
             # provider timeouts, 5xx, etc.) before the agent surfaces the
             # failure.  The OpenAI SDK already does its own low-level retries

  • tests/gateway/test_gateway_shutdown.py

    @@ -94,6 +94,10 @@ async def block_forever(_event):
     @pytest.mark.asyncio
     async def test_gateway_stop_drains_running_agents_before_disconnect():
         runner, adapter = make_restart_runner()
    +    # Opt into a grace window (the default is 0 = interrupt immediately).
    +    # This exercises the path where an agent finishes within the drain
    +    # window and must NOT be interrupted.
    +    runner._restart_drain_timeout = 5.0
         disconnect_mock = AsyncMock()
         adapter.disconnect = disconnect_mock