# NousResearch/hermes-agent · Gateway platform reconnect retry reliability fixes ## Open now - Nothing open on GitHub. These issues and PRs address problems where the gateway's reconnect watcher would permanently stop retrying platform connections after a limited number of failures or prolonged network loss. The work focuses on ensuring retryable platforms remain queued and reconnect attempts continue indefinitely through transient network and DNS failures. ## TL;DR - 7 member(s): 0 open issue(s), 2 closed issue(s), 0 open PR(s), 2 merged PR(s), 3 PR(s) closed without merging. - A fix was merged to main in PR #35387. Verify it addresses your case. - **Recommended:** read PR #35387 (and any competing PRs below) before filing. Confirm the merged change covers your symptom. ## Maintainer resolution verdict - Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-26600`. This is distinct from the fix status above. Weigh it yourself. ## How this cluster was grouped - Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text. - Members outside that structural core may belong to a different underlying problem. - Pairwise link density `0.26`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked. ## Closed / duplicate issues - #12607: fix(gateway): retryable reconnects stop after prolonged network loss - #17063: Gateway reconnect watcher permanently stops retryable platforms after 20 failed attempts ## Prior & competing PRs - **#26600: fix(gateway): keep running when platforms fail; add per-platform circuit breaker + /platform**. merged to main · fix tier `suggested` - maintainer direction: category `merged`; implication `superseded` - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._ - **#35387: fix(gateway): never auto-pause platforms on transient network/DNS failures**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `gateway/run.py`, `tests/gateway/test_platform_reconnect.py` - fix hunks: - `gateway/run.py` ```diff @@ -1734,9 +1734,9 @@ def __init__(self, config: Optional[GatewayConfig] = None): # as a fallback when a fresh config read transiently returns an empty # model (e.g. an mtime-keyed config-cache miss during a post-interrupt # recovery turn). Without this, the agent is built with model="" and - # every API call fails HTTP 400 "No models provided" — the session - # goes silent until the user manually re-sends. See #35314. The "*" - # key holds a process-wide last-known-good for first-seen sessions. + # every API call fails HTTP 400 "No models provided" — the session goes + # silent until the user manually re-sends. See #35314. ``"*"`` holds a + # process-wide last-known-good for sessions seen for the first time. self._last_resolved_model: Dict[str, str] = {} # Overflow buffer for explicit /queue commands. The adapter-level # _pending_messages dict is a single slot per session (designed for ``` ```diff @@ -2502,8 +2502,8 @@ def _resolve_session_agent_runtime( # successfully resolved for this session (or, failing that, the most # recent one resolved process-wide). Building an agent with model="" # makes every API call fail HTTP 400 "No models provided" and the - # session goes silent until the user manually re-sends. getattr guards - # against bare test runners built via object.__new__. + # session goes silent until the user manually re-sends. ``getattr`` + # guards against bare test runners built via ``object.__new__``. _last_good = getattr(self, "_last_resolved_model", None) if _last_good is not None: if not model: ``` ```diff @@ -2818,10 +2818,12 @@ def _pause_failed_platform(self, platform, *, reason: str = "") -> None: """Mark a queued platform as paused — keep it in ``_failed_platforms`` but stop the reconnect watcher from hammering it. - Used by the circuit breaker after ``_PAUSE_AFTER_FAILURES`` consecutive - retryable failures, and by ``/platform pause `` for manual - intervention. Paused platforms are surfaced in ``/platform list`` - and resumed with ``/platform resume ``. + Used by ``/platform pause `` for manual operator intervention. + Paused platforms are surfaced in ``/platform list`` and resumed with + ``/platform resume ``. Note: the reconnect watcher does NOT + auto-pause — retryable (network/DNS) failures keep retrying at the + backoff cap indefinitely so a transient outage self-heals without + manual intervention. """ info = getattr(self, "_failed_platforms", {}).get(platform) if info is None: ``` ```diff @@ -5899,15 +5901,17 @@ async def _platform_reconnect_watcher(self) -> None: """Background task that periodically retries connecting failed platforms. Uses exponential backoff: 30s → 60s → 120s → 240s → 300s (cap). - Retryable failures keep retrying at the backoff cap indefinitely - — but if a platform fails ``_PAUSE_AFTER_FAILURES`` times in a row - without ever succeeding, it is *paused*: kept in the retry queue - but no longer hammered. The user surfaces it with ``/platform list`` - and resumes it with ``/platform resume ``. Non-retryable - failures (bad auth, etc.) still drop out of the queue immediately. + Retryable failures (network/DNS blips) keep retrying at the backoff + cap indefinitely — they self-heal once connectivity returns, so a + transient outage never requires manual intervention. Non-retryable + failures (bad auth, etc.) drop out of the queue immediately. The + circuit breaker (``_pause_failed_platform`` / ``/platform pause``) + remains available for manual operator control via ``/platform list`` + and ``/platform resume ``, but is no longer triggered + automatically — auto-pausing a recovered platform was the cause of + bots silently staying dead after a transient DNS failure. """ _BACKOFF_CAP = 300 # 5 minutes max between retries - _PAUSE_AFTER_FAILURES = 10 # circuit-breaker threshold await asyncio.sleep(10) # initial delay — let startup finish while self._running: ``` ```diff @@ -6002,14 +6006,14 @@ async def _platform_reconnect_watcher(self) -> None: "Reconnect %s failed, next retry in %ds", platform.value, backoff, ) - if attempt >= _PAUSE_AFTER_FAILURES: - self._pause_failed_platform( - platform, - reason=( - adapter.fatal_error_message - or "failed to reconnect" - ), - ) + # Retryable failures (network/DNS blips) keep retrying + # at the backoff cap indefinitely — they self-heal once + # connectivity returns. We do NOT auto-pause them: a + # transient outage must never require manual `/platform + # resume` to recover. Non-retryable failures (bad auth, + # etc.) already drop out of the queue via the + # `not fatal_error_retryable` branch above, so anything + # reaching here is by definition retryable. except Exception as e: self._update_platform_runtime_status( platform.value, ``` ```diff @@ -6024,8 +6028,9 @@ async def _platform_reconnect_watcher(self) -> None: "Reconnect %s error: %s, next retry in %ds", platform.value, e, backoff, ) - if attempt >= _PAUSE_AFTER_FAILURES: - self._pause_failed_platform(platform, reason=str(e)) + # A raised exception during reconnect (connect timeout, DNS + # resolution failure, etc.) is inherently transient — keep + # retrying at the backoff cap rather than auto-pausing. # Check every 10 seconds for platforms that need reconnection for _ in range(10): ``` - `tests/gateway/test_platform_reconnect.py` ```diff @@ -294,19 +294,20 @@ async def fake_sleep(n): assert runner._failed_platforms[Platform.TELEGRAM]["attempts"] == 2 @pytest.mark.asyncio - async def test_reconnect_pauses_after_circuit_breaker_threshold(self): - """After enough consecutive retryable failures, the watcher should - *pause* the platform (keep it in the queue but stop hammering it), - not drop it. The user resumes via /platform resume. + async def test_reconnect_never_auto_pauses_retryable_failures(self): + """Retryable failures (network/DNS) must keep retrying indefinitely — + the watcher must NOT auto-pause them. Auto-pausing a transiently-failed + platform left bots silently dead after a DNS blip (#35284). The pause + circuit breaker remains available for manual /platform pause only. """ runner = _make_runner() platform_config = PlatformConfig(enabled=True, token="test") - # 9 prior attempts — the next failure will be the 10th and should - # trip the circuit breaker. + # Far past the old circuit-breaker threshold (10): even after many + # consecutive retryable failures the platform must stay unpaused. runner._failed_platforms[Platform.TELEGRAM] = { "config": platform_config, - "attempts": 9, + "attempts": 25, "next_retry": time.monotonic() - 1, } ``` ```diff @@ -332,12 +333,15 @@ async def fake_sleep(n): await run_one_iteration() - # Platform stays in queue — paused, not dropped + # Platform stays in queue and keeps retrying — never auto-paused. assert Platform.TELEGRAM in runner._failed_platforms info = runner._failed_platforms[Platform.TELEGRAM] - assert info["paused"] is True - assert info["attempts"] == 10 - assert "pause_reason" in info + assert info.get("paused") is not True + assert "pause_reason" not in info + assert info["attempts"] == 26 + # next_retry is pushed out by the backoff (capped at 300s), not inf. + assert info["next_retry"] != float("inf") + assert info["next_retry"] > time.monotonic() @pytest.mark.asyncio async def test_reconnect_skips_paused_platforms(self): ``` - **#13197: fix(gateway): keep retryable platform reconnects queued**. closed without merge - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `45465b0d5`, `#35387` - **#17216: fix: gateway reconnect watcher retries indefinitely instead of giving up after 20 attempts**. closed without merge - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `45465b0d5d8c7b2db7df6d9e466589cdef9136c0`, `#35387` - **#17219: fix(gateway): don't give up on retryable platform reconnect (#17063)**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#17063`, `#26600` ## Fix status - Merged to main in PR #35387 (cited above). No release data is tracked. Verify downstream. - Fix last verified (merged) `2026-05-30`. - Subsystem: `gateway`