# NousResearch/hermes-agent · Gateway resource/FD leaks during reconnect and disconnect ## Open now - Nothing open on GitHub. Multiple fixes targeting file descriptor and SQLite connection leaks in the gateway's ResponseStore and adapter resources, particularly during disconnect and unsuccessful reconnect scenarios. One related change adds a requirement for API_SERVER_KEY when loading the api_server platform. ## TL;DR - 7 member(s): 0 open issue(s), 1 closed issue(s), 0 open PR(s), 3 merged PR(s), 3 PR(s) closed without merging. - A fix was merged to main in PR #56260. Verify it addresses your case. - **Recommended:** read PR #56260 (and any competing PRs below) before filing. Confirm the merged change covers your symptom. ## Maintainer resolution verdict - Internal-tracker verdict over this cluster's issue members: `unresolved`. This is distinct from the fix status above. Weigh it yourself. ## How this cluster was grouped - Link evidence (`STRUCTURAL_SPANNING`): every member is connected by structural evidence: shared changed files, or a closing/superseded reference. - Pairwise link density `0.29`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked. ## Closed / duplicate issues - #36111: File descriptor leak in api_server platform: ResponseStore SQLite connections not closed on retry ## Prior & competing PRs - **#37389: fix: ResponseStore SQLite connection pooling to prevent FD leak (#36111, #37369)**. closed without merge · fix tier `suggested` - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `4b06c98fe4f4686f89873933ac616cb6e76b298d` - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._ - **#56260: fix(api-server): require auth for /health/detailed and fail closed on weak keys**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `gateway/platforms/api_server.py`, `tests/gateway/test_api_server.py`, `tests/gateway/test_api_server_bind_guard.py` - fix hunks: - `gateway/platforms/api_server.py` ```diff @@ -1165,8 +1165,12 @@ async def _handle_health_detailed(self, request: "web.Request") -> "web.Response Returns gateway state, connected platforms, PID, and uptime so the dashboard can display full status without needing a shared PID file or - /proc access. No authentication required. + /proc access. Requires the same Bearer auth as other API routes. """ + auth_err = self._check_auth(request) + if auth_err: + return auth_err + from gateway.status import ( derive_gateway_busy, derive_gateway_drainable, ``` ```diff @@ -4454,12 +4458,60 @@ async def _sweep_orphaned_runs(self) -> None: # BasePlatformAdapter interface # ------------------------------------------------------------------ + def _api_key_passes_startup_guard(self) -> bool: + """Return True when API_SERVER_KEY is present and strong enough to start.""" + if not self._api_key: + logger.error( + "[%s] Refusing to start: API_SERVER_KEY is required for the API server, " + "including loopback-only binds on %s.", + self.name, self._host, + ) + return False + + try: + from hermes_cli.auth import has_usable_secret + if not has_usable_secret(self._api_key, min_length=16): + logger.error( + "[%s] Refusing to start: API_SERVER_KEY is a " + "placeholder or too short (<16 chars). This endpoint " + "dispatches terminal-capable agent work — a guessable " + "key is remote code execution. Generate a strong secret " + "(e.g. `openssl rand -hex 32`) and set API_SERVER_KEY " + "before starting the API server on %s.", + self.name, self._host, + ) + return False + except ImportError: + pass + return True + + def _port_is_available(self) -> bool: + """Return True when the configured listen port is free.""" + try: + with _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM) as _s: + _s.settimeout(1) + _s.connect(('127.0.0.1', self._port)) + logger.error( + "[%s] Port %d already in use. Set a different port in config.yaml: " + "platforms.api_server.port", + self.name, self._port, + ) + return False + except (ConnectionRefusedError, OSError): + return True + async def connect(self, *, is_reconnect: bool = False) -> bool: """Start the aiohttp web server.""" if not AIOHTTP_AVAILABLE: logger.warning("[%s] aiohttp not installed", self.name) return False + if not self._api_key_passes_startup_guard(): + return False + + if not self._port_is_available(): + return False + try: mws = [mw for mw in (cors_middleware, body_limit_middleware, security_headers_middleware) if mw is not None] self._app = web.Application(middlewares=mws, client_max_size=MAX_REQUEST_BYTES) ``` ```diff @@ -4520,39 +4572,6 @@ async def connect(self, *, is_reconnect: bool = False) -> bool: if hasattr(sweep_task, "add_done_callback"): sweep_task.add_done_callback(self._background_tasks.discard) - # Refuse to start without authentication. The API server can - # dispatch terminal-capable agent work, so every deployment needs - # an explicit API_SERVER_KEY regardless of bind address. - if not self._api_key: - logger.error( - "[%s] Refusing to start: API_SERVER_KEY is required for the API server, " - "including loopback-only binds on %s.", - self.name, self._host, - ) - return False - - # Refuse to start network-accessible with a placeholder or weak key. - # Ported from openclaw/openclaw#64586; entropy floor raised to 16 in - # the June 2026 hermes-0day hardening (an 8-char key dispatching - # terminal-capable agent work on a public bind is brute-forceable). - if is_network_accessible(self._host) and self._api_key: - try: - from hermes_cli.auth import has_usable_secret - if not has_usable_secret(self._api_key, min_length=16): - logger.error( - "[%s] Refusing to start: API_SERVER_KEY is a " - "placeholder or too short (<16 chars) for a " - "network-accessible bind. This endpoint dispatches " - "terminal-capable agent work — a guessable key is " - "remote code execution. Generate a strong secret " - "(e.g. `openssl rand -hex 32`) and set " - "API_SERVER_KEY before exposing it on %s.", - self.name, self._host, - ) - return False - except ImportError: - pass - # Loud warning when a network-accessible API server runs against an # unsandboxed local terminal backend. The API server can drive the # agent's terminal/file tools as the host user; on a public bind ``` ```diff @@ -4581,16 +4600,6 @@ async def connect(self, *, is_reconnect: bool = False) -> bool: self.name, self._host, ) - # Port conflict detection — fail fast if port is already in use - try: - with _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM) as _s: - _s.settimeout(1) - _s.connect(('127.0.0.1', self._port)) - logger.error('[%s] Port %d already in use. Set a different port in config.yaml: platforms.api_server.port', self.name, self._port) - return False - except (ConnectionRefusedError, OSError): - pass # port is free - self._runner = web.AppRunner(self._app) await self._runner.setup() self._site = web.TCPSite(self._runner, self._host, self._port) ``` - `tests/gateway/test_api_server.py` ```diff @@ -772,12 +772,21 @@ async def test_health_detailed_no_runtime_status(self, adapter): assert data["gateway_drainable"] is False @pytest.mark.asyncio - async def test_health_detailed_does_not_require_auth(self, auth_adapter): - """Health detailed endpoint should be accessible without auth, like /health.""" + async def test_health_detailed_requires_auth(self, auth_adapter): + """Detailed health must not leak runtime state without Bearer auth.""" app = _create_app(auth_adapter) with patch("gateway.status.read_runtime_status", return_value=None): async with TestClient(TestServer(app)) as cli: resp = await cli.get("/health/detailed") + assert resp.status == 401 + + @pytest.mark.asyncio + async def test_health_detailed_allows_authenticated_request(self, auth_adapter): + app = _create_app(auth_adapter) + headers = {"Authorization": f"Bearer {auth_adapter._api_key}"} + with patch("gateway.status.read_runtime_status", return_value={"gateway_state": "running"}): + async with TestClient(TestServer(app)) as cli: + resp = await cli.get("/health/detailed", headers=headers) assert resp.status == 200 ``` - `tests/gateway/test_api_server_bind_guard.py` ```diff @@ -119,6 +119,19 @@ async def test_refuses_loopback_without_key(self): assert is_network_accessible(adapter._host) is False result = await adapter.connect() assert result is False + assert adapter._app is None + assert adapter._background_tasks == set() + + @pytest.mark.asyncio + async def test_refuses_weak_key_without_partial_startup(self): + """Weak API_SERVER_KEY rejection must not create app or background tasks.""" + adapter = APIServerAdapter( + PlatformConfig(enabled=True, extra={"host": "127.0.0.1", "key": "short"}), + ) + result = await adapter.connect() + assert result is False + assert adapter._app is None + assert adapter._background_tasks == set() @pytest.mark.asyncio async def test_allows_wildcard_with_key(self): ``` - **#69838: fix(api_server): fail closed when API_SERVER_KEY strength can't be verified**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `gateway/platforms/api_server.py`, `tests/gateway/test_api_server.py` - **#70166: fix(gateway): require usable API_SERVER_KEY to enroll the api_server platform (#36180 salvage)**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `contributors/emails/[redacted:email]`, `gateway/config.py`, `tests/gateway/test_api_server.py`, `tests/gateway/test_platform_connected_checkers.py` - **#36180: fix(gateway): require API_SERVER_KEY to load the api_server platform**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `#70166`, `#36111` - **#36242: fix(gateway): close api server response store leak**. closed without merge - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `4b06c98fe4`, `#56260`, `#69838`, `#36180`, `#36111` ## Fix status - Merged to main in PR #56260 (cited above). No release data is tracked. Verify downstream. - Fix last verified (merged) `2026-07-23`. - Subsystem: `gateway`