{"repo":"NousResearch/hermes-agent","public_id":"42072","display_label":"MCP stdio subprocess memory leak and recycling fixes","summary":"Long-lived gateway processes (e.g. Playwright Chromium) were never recycled, causing significant memory growth over time. These members cover the reported leak and fixes targeting idle recycling and subprocess lifecycle management.","subsystem":"mcp","counts":{"members":4,"open_issues":0,"open_issues_distinct":0,"open_prs":0,"closed_issues":2,"merged_prs":1,"attempt_prs":1},"landed_fix":{"status":"merged_to_main","basis":"structural","pr":60509},"resolution":{"state":"unresolved","by":null},"freshness":{"state":"verified","last_verified_at":"2026-07-07T22:16:01+00:00","is_stale":false},"grouping":{"backbone":"STRUCTURAL_PARTIAL","link_density":0.3933333333333333},"fix_edges":[{"issue_number":42073,"pr_number":60509,"tier":"verified","basis":"superseded_ref","confidence":0.9,"recoverable_from_timeline":false},{"issue_number":57228,"pr_number":60509,"tier":"verified","basis":"superseded_ref","confidence":0.9,"recoverable_from_timeline":false},{"issue_number":57355,"pr_number":60509,"tier":"verified","basis":"superseded_ref","confidence":0.9,"recoverable_from_timeline":false},{"issue_number":59349,"pr_number":60509,"tier":"verified","basis":"superseded_ref","confidence":0.9,"recoverable_from_timeline":false}],"members":[{"kind":"issue","number":42072,"title":"MCP stdio subprocesses never recycled — long-lived gateway leaks memory (Playwright Chromium ~740MB in 1.5 days)","url":null,"state":"closed"},{"kind":"issue","number":42073,"title":"MCP stdio subprocesses never recycled — long-lived gateway leaks memory (Playwright Chromium 740MB in 1.5 days)","url":null,"state":"closed"},{"kind":"pr","number":42311,"title":"fix(mcp): recycle idle stdio servers","url":null,"state":"closed","close_reason":{"refs":[{"kind":"pr","value":"#60509","number":60509},{"kind":"pr","value":"#42073","number":42073}],"category":"merged_via","implication":"superseded"},"merged":false},{"kind":"pr","number":60509,"title":"fix(mcp): close all four stdio subprocess leak paths (reconnect zombies, hung handshake, parent death, idle residency)","url":null,"state":"merged","close_reason":{"category":"merged","implication":"superseded"},"merged":true,"is_fix_pr":true,"fix_tier":"verified","files_changed":["scripts/release.py","tests/tools/test_mcp_circuit_breaker.py","tests/tools/test_mcp_stability.py","tests/tools/test_mcp_stdio_init_timeout.py","tests/tools/test_mcp_tool.py","tests/tools/test_mcp_tool_issue_948.py","tools/mcp_stdio_watchdog.py","tools/mcp_tool.py","website/docs/user-guide/features/mcp.md"],"fix_hunks":[{"file":"scripts/release.py","hunks":["@@ -45,6 +45,8 @@\n \n # Auto-extracted from noreply emails + manual overrides\n AUTHOR_MAP = {\n+    \"[redacted:email]\": \"rainbowgore\",  # PR #59405 salvage (mcp: bound stdio initialize handshake to stop subprocess/FD leak; #59349)\n+    \"[redacted:email]\": \"thestudionorth\",  # PR #60015 salvage (mcp: parent-death watchdog for stdio children; commit under unlinked local identity)\n     \"[redacted:email]\": \"vampyren\",  # PR #59830 salvage (kanban: grab-to-pan board scrolling; original commit under unlinked local identity)\n     \"[redacted:email]\": \"spiky02plateau\",  # PR #32824 salvage (usage: fetch Codex account limits from the credential pool in pool-only setups; superseded by #60028)\n     \"[redacted:email]\": \"hwrdprkns\",  # PR #36896 salvage (secrets: [redacted:secret-assignment] op:// secret source + shared _cache substrate, adapted onto the SecretSource interface)"]},{"file":"tests/tools/test_mcp_circuit_breaker.py","hunks":["@@ -74,6 +74,13 @@ def assert_called_once(self):\n \n     server._reconnect_event = _ReconnectAdapter()\n     server._ready = _ReadyAdapter()\n+    # A bare MagicMock returns a truthy Mock for every method, so\n+    # ``_is_recycled_stdio()`` would spuriously report this stub as a recycled\n+    # stdio server and divert dead-session tool calls into the lazy-reconnect\n+    # wait (which polls the test-frozen ``time.monotonic`` forever). Real\n+    # non-recycled servers return False here; make the stub faithful so the\n+    # dead-session path falls through to the graceful reconnect handler.\n+    server._is_recycled_stdio.return_value = False\n \n     mcp_tool_module._servers[name] = server\n     mcp_tool_module._server_error_counts.pop(name, None)"]},{"file":"tests/tools/test_mcp_stability.py","hunks":["@@ -109,6 +109,7 @@ def test_kill_orphaned_noop_when_empty(self):\n         \"\"\"_kill_orphaned_mcp_children does nothing when no PIDs tracked.\"\"\"\n         from tools.mcp_tool import (\n             _kill_orphaned_mcp_children,\n+            _orphan_stdio_pid_servers,\n             _orphan_stdio_pids,\n             _stdio_pids,\n             _lock,","@@ -117,6 +118,7 @@ def test_kill_orphaned_noop_when_empty(self):\n         with _lock:\n             _stdio_pids.clear()\n             _orphan_stdio_pids.clear()\n+            _orphan_stdio_pid_servers.clear()\n \n         # Should not raise\n         _kill_orphaned_mcp_children()","@@ -125,6 +127,7 @@ def test_kill_orphaned_handles_dead_pids(self):\n         \"\"\"_kill_orphaned_mcp_children gracefully handles already-dead PIDs.\"\"\"\n         from tools.mcp_tool import (\n             _kill_orphaned_mcp_children,\n+            _orphan_stdio_pid_servers,\n             _orphan_stdio_pids,\n             _lock,\n         )","@@ -133,6 +136,7 @@ def test_kill_orphaned_handles_dead_pids(self):\n         fake_pid = 999999999\n         with _lock:\n             _orphan_stdio_pids.add(fake_pid)\n+            _orphan_stdio_pid_servers[fake_pid] = \"orphan\"\n \n         # Should not raise (ProcessLookupError is caught)\n         _kill_orphaned_mcp_children()","@@ -144,6 +148,7 @@ def test_kill_orphaned_uses_sigkill_when_available(self, monkeypatch):\n         \"\"\"SIGTERM-first then SIGKILL after 2s for orphan cleanup.\"\"\"\n         from tools.mcp_tool import (\n             _kill_orphaned_mcp_children,\n+            _orphan_stdio_pid_servers,\n             _orphan_stdio_pids,\n             _lock,\n         )","@@ -151,7 +156,9 @@ def test_kill_orphaned_uses_sigkill_when_available(self, monkeypatch):\n         fake_pid = 424242\n         with _lock:\n             _orphan_stdio_pids.clear()\n+            _orphan_stdio_pid_servers.clear()\n             _orphan_stdio_pids.add(fake_pid)\n+            _orphan_stdio_pid_servers[fake_pid] = \"orphan\"\n \n         fake_sigkill = 9\n         monkeypatch.setattr(signal, \"SIGKILL\", fake_sigkill, raising=False)","@@ -177,6 +184,7 @@ def test_kill_orphaned_falls_back_without_sigkill(self, monkeypatch):\n         \"\"\"Without SIGKILL, SIGTERM is used for both phases.\"\"\"\n         from tools.mcp_tool import (\n             _kill_orphaned_mcp_children,\n+            _orphan_stdio_pid_servers,\n             _orphan_stdio_pids,\n             _lock,\n         )","@@ -184,7 +192,9 @@ def test_kill_orphaned_falls_back_without_sigkill(self, monkeypatch):\n         fake_pid = 434343\n         with _lock:\n             _orphan_stdio_pids.clear()\n+            _orphan_stdio_pid_servers.clear()\n             _orphan_stdio_pids.add(fake_pid)\n+            _orphan_stdio_pid_servers[fake_pid] = \"orphan\"\n \n         monkeypatch.delattr(signal, \"SIGKILL\", raising=False)\n ","@@ -199,6 +209,99 @@ def test_kill_orphaned_falls_back_without_sigkill(self, monkeypatch):\n         with _lock:\n             assert fake_pid not in _orphan_stdio_pids\n \n+    def test_run_stdio_reaps_orphans_before_spawn(self):\n+        \"\"\"_run_stdio kills orphaned PIDs from prior failed attempts (#57355).\"\"\"\n+        from tools.mcp_tool import (\n+            _kill_orphaned_mcp_children,\n+            _orphan_stdio_pids,\n+            _stdio_pids,\n+            _stdio_pgids,\n+            _lock,\n+            MCPServerTask,\n+        )\n+        from unittest.mock import patch, MagicMock, AsyncMock\n+\n+        # Seed an orphan PID that belongs to a prior failed connection.\n+        fake_pid = 999999997\n+        with _lock:\n+            _orphan_stdio_pids.add(fake_pid)\n+\n+        server = MCPServerTask.__new__(MCPServerTask)\n+        server.name = \"test-zombie-reap\"\n+        server._ready = MagicMock()\n+        server._shutdown_event = MagicMock()\n+        server._shutdown_event.is_set.return_value = True\n+        server._reconnect_event = MagicMock()\n+        server._sampling = None\n+        server._elicitation = None\n+        server._registered_tool_names = []\n+\n+        config = {\"command\": \"echo\", \"args\": [\"hello\"]}\n+\n+        import asyncio\n+\n+        async def _run():\n+            # _run_stdio should reap orphans before it gets to the\n+            # stdio_client spawn.  Patch the OSV check (local import)\n+            # and stdio_client so no real subprocess is spawned.\n+            with patch(\"tools.mcp_tool._MCP_AVAILABLE\", True), \\\n+                 patch(\"tools.mcp_tool._build_safe_env\", return_value={}), \\\n+                 patch(\"tools.mcp_tool._resolve_stdio_command\",\n+                       return_value=(\"echo\", {})), \\\n+                 patch(\"tools.mcp_tool._write_stderr_log_header\"), \\\n+                 patch(\"tools.mcp_tool._get_mcp_stderr_log\",\n+                       return_value=None), \\\n+                 patch(\"tools.mcp_tool.check_package_for_malware\",\n+                       return_value=None, create=True), \\\n+                 patch(\"tools.osv_check.check_package_for_malware\",\n+                       return_value=None):\n+                # Patch stdio_client to raise so the test exits quickly\n+                cm = MagicMock()\n+                cm.__aenter__ = AsyncMock(side_effect=RuntimeError(\"test\"))\n+                cm.__aexit__ = AsyncMock(return_value=False)\n+                with patch(\"tools.mcp_tool.stdio_client\", return_value=cm):\n+                    try:\n+                        await server._run_stdio(config)\n+                    except Exception:\n+                        pass\n+\n+        asyncio.run(_run())\n+\n+        # The orphan must have been reaped before the spawn attempt.\n+        with _lock:\n+            assert fake_pid not in _orphan_stdio_pids\n+\n+    def test_kill_orphaned_can_filter_by_server_name(self):\n+        \"\"\"Reconnect cleanup reaps only the orphan owned by that MCP server.\"\"\"\n+        from tools.mcp_tool import (\n+            _kill_orphaned_mcp_children,\n+            _orphan_stdio_pid_servers,\n+            _orphan_stdio_pids,\n+            _lock,\n+        )\n+\n+        target_pid = 454545\n+        other_pid = 464646\n+        with _lock:\n+            _orphan_stdio_pids.clear()\n+            _orphan_stdio_pid_servers.clear()\n+            _orphan_stdio_pids.update({target_pid, other_pid})\n+            _orphan_stdio_pid_servers[target_pid] = \"feishu\"\n+            _orphan_stdio_pid_servers[other_pid] = \"mimir\"\n+\n+        with patch(\"tools.mcp_tool.os.kill\") as mock_kill, \\\n+             patch(\"gateway.status._pid_exists\", return_value=False), \\\n+             patch(\"tools.mcp_tool.time.sleep\") as mock_sleep:\n+            _kill_orphaned_mcp_children(server_name=\"feishu\")\n+\n+        mock_kill.assert_called_once_with(target_pid, signal.SIGTERM)\n+        mock_sleep.assert_called_once_with(2)\n+        with _lock:\n+            assert target_pid not in _orphan_stdio_pids\n+            assert target_pid not in _orphan_stdio_pid_servers\n+            assert other_pid in _orphan_stdio_pids\n+            assert _orphan_stdio_pid_servers[other_pid] == \"mimir\"\n+\n \n # ---------------------------------------------------------------------------\n # Fix 2b: stdio descendant reaping via process group (issue #23799)","@@ -214,10 +317,17 @@ class TestStdioPgroupReaping:\n     \"\"\"_kill_orphaned_mcp_children reaps via killpg when a pgid is tracked.\"\"\"\n \n     def _reset_state(self):\n-        from tools.mcp_tool import _stdio_pids, _orphan_stdio_pids, _stdio_pgids, _lock\n+        from tools.mcp_tool import (\n+            _orphan_stdio_pid_servers,\n+            _orphan_stdio_pids,\n+            _stdio_pgids,\n+            _stdio_pids,\n+            _lock,\n+        )\n         with _lock:\n             _stdio_pids.clear()\n             _orphan_stdio_pids.clear()\n+            _orphan_stdio_pid_servers.clear()\n             _stdio_pgids.clear()\n \n     def test_killpg_used_when_pgid_tracked(self, monkeypatch):","@@ -443,6 +553,7 @@ def test_grandchild_reaped_via_pgroup(self, tmp_path):\n         # Drive the reaper: register the parent pid + pgid as an orphan.\n         from tools.mcp_tool import (\n             _kill_orphaned_mcp_children,\n+            _orphan_stdio_pid_servers,\n             _orphan_stdio_pids,\n             _stdio_pgids,\n             _stdio_pids,","@@ -451,8 +562,10 @@ def test_grandchild_reaped_via_pgroup(self, tmp_path):\n         with _lock:\n             _stdio_pids.clear()\n             _orphan_stdio_pids.clear()\n+            _orphan_stdio_pid_servers.clear()\n             _stdio_pgids.clear()\n             _orphan_stdio_pids.add(parent.pid)\n+            _orphan_stdio_pid_servers[parent.pid] = \"orphan\"\n             _stdio_pgids[parent.pid] = parent_pgid\n         try:\n             _kill_orphaned_mcp_children()"]},{"file":"tests/tools/test_mcp_stdio_init_timeout.py","hunks":["@@ -0,0 +1,94 @@\n+\"\"\"Regression test for the stdio-MCP subprocess/FD leak (#59349).\n+\n+A stdio MCP server that never completes ``initialize`` (e.g. emits a\n+non-JSON-RPC frame and then blocks on stdin) used to hang ``_run_stdio``\n+forever on the background event loop: ``connect_timeout`` bounded only the\n+*caller's* ``.result()`` wait, not the coroutine itself. Because the connect\n+never unwound, the cleanup ``finally`` in ``_run_stdio`` never ran, so the\n+spawned child process and its stdio pipes / pidfd leaked on *every* discovery\n+retry — unbounded, until the gateway hit EMFILE.\n+\n+The fix wraps ``session.initialize()`` in\n+``asyncio.wait_for(..., timeout=connect_timeout)`` so a stalled handshake fails\n+instead of hanging, which lets the existing ``finally`` reap the child.\n+\n+This test drives the *real* ``_run_stdio`` with a fake transport whose\n+``initialize()`` hangs, and asserts the connect is bounded by\n+``connect_timeout`` rather than blocking forever. It is fully hermetic — no real\n+subprocess, no network (the drain-to-zero behaviour was additionally verified\n+manually against the reporter's live repro).\n+\"\"\"\n+\n+from __future__ import annotations\n+\n+import asyncio\n+import time\n+from unittest.mock import patch\n+\n+import pytest\n+\n+pytest.importorskip(\"mcp\")\n+\n+\n+class _HangingSession:\n+    \"\"\"Stand-in ClientSession whose handshake never completes.\"\"\"\n+\n+    async def initialize(self):\n+        await asyncio.sleep(3600)\n+\n+\n+class _FakeAsyncCM:\n+    \"\"\"Minimal async context manager yielding a fixed value; spawns nothing.\"\"\"\n+\n+    def __init__(self, value):\n+        self._value = value\n+\n+    async def __aenter__(self):\n+        return self._value\n+\n+    async def __aexit__(self, *_exc):\n+        return False\n+\n+\n+def _fake_stdio_client(*_args, **_kwargs):\n+    # `async with stdio_client(...) as (read, write)` — no subprocess spawned.\n+    return _FakeAsyncCM((object(), object()))\n+\n+\n+def _fake_client_session(*_args, **_kwargs):\n+    # `async with ClientSession(...) as session` -> a session that hangs.\n+    return _FakeAsyncCM(_HangingSession())\n+\n+\n+class TestStdioInitializeTimeout:\n+    def test_hanging_initialize_is_bounded_not_leaked(self):\n+        \"\"\"A stdio server that hangs at ``initialize`` must fail within\n+        ``connect_timeout`` — not block ``_run_stdio`` forever (#59349).\"\"\"\n+        from tools import mcp_tool\n+\n+        server = mcp_tool.MCPServerTask(\"leak-guard\")\n+        config = {\"command\": \"fake-mcp\", \"args\": [], \"connect_timeout\": 0.2}\n+\n+        async def drive():\n+            with patch.object(mcp_tool, \"stdio_client\", _fake_stdio_client), \\\n+                 patch.object(mcp_tool, \"ClientSession\", _fake_client_session), \\\n+                 patch.object(mcp_tool, \"_resolve_stdio_command\", lambda c, e: (c, e)), \\\n+                 patch.object(mcp_tool, \"_write_stderr_log_header\", lambda *_a, **_k: None), \\\n+                 patch.object(mcp_tool, \"_get_mcp_stderr_log\", lambda: None), \\\n+                 patch(\"tools.osv_check.check_package_for_malware\",\n+                       lambda *_a, **_k: None):\n+                start = time.monotonic()\n+                # The outer 5s guard exists ONLY so a regression can't hang the\n+                # whole suite. With the fix, the inner connect_timeout (0.2s)\n+                # fires first; the elapsed assertion below is what actually\n+                # distinguishes \"bounded\" (fixed) from \"hung\" (regressed).\n+                with pytest.raises(asyncio.TimeoutError):\n+                    await asyncio.wait_for(server._run_stdio(config), timeout=5.0)\n+                return time.monotonic() - start\n+\n+        elapsed = asyncio.run(drive())\n+        assert elapsed < 2.0, (\n+            f\"_run_stdio blocked {elapsed:.1f}s on a hanging initialize() — the \"\n+            f\"connect_timeout ({config['connect_timeout']}s) bound was not applied; \"\n+            f\"the #59349 subprocess/FD leak has regressed.\"\n+        )"]},{"file":"tests/tools/test_mcp_tool.py","hunks":["@@ -172,6 +172,56 @@ def test_status_distinguishes_configured_connecting_failed_and_disabled(\n         assert statuses[\"disabled\"][\"disabled\"] is True\n \n \n+class TestLifecycleConfig:\n+    def test_get_lifecycle_seconds_accepts_top_level_and_nested_values(self):\n+        from tools.mcp_tool import _get_lifecycle_seconds\n+\n+        assert (\n+            _get_lifecycle_seconds(\n+                {\"idle_timeout_seconds\": \"3.5\"},\n+                \"idle_timeout_seconds\",\n+            )\n+            == 3.5\n+        )\n+        assert _get_lifecycle_seconds(\n+            {\"lifecycle\": {\"max_lifetime_seconds\": 42}},\n+            \"max_lifetime_seconds\",\n+        ) == 42.0\n+\n+    def test_get_lifecycle_seconds_treats_zero_as_disabled(self):\n+        from tools.mcp_tool import _get_lifecycle_seconds\n+\n+        assert (\n+            _get_lifecycle_seconds(\n+                {\"idle_timeout_seconds\": 0},\n+                \"idle_timeout_seconds\",\n+            )\n+            is None\n+        )\n+\n+    def test_get_lifecycle_seconds_ignores_invalid_values(self, caplog):\n+        from tools.mcp_tool import _get_lifecycle_seconds\n+\n+        assert (\n+            _get_lifecycle_seconds(\n+                {\"idle_timeout_seconds\": \"soon\"},\n+                \"idle_timeout_seconds\",\n+            )\n+            is None\n+        )\n+        assert (\n+            _get_lifecycle_seconds(\n+                {\"idle_timeout_seconds\": -1},\n+                \"idle_timeout_seconds\",\n+            )\n+            is None\n+        )\n+\n+        messages = [record.getMessage() for record in caplog.records]\n+        assert any(\"must be a number of seconds\" in msg for msg in messages)\n+        assert any(\"must be positive\" in msg for msg in messages)\n+\n+\n # ---------------------------------------------------------------------------\n # Schema conversion\n # ---------------------------------------------------------------------------","@@ -568,6 +618,19 @@ def test_session_none_returns_false(self):\n         finally:\n             _servers.pop(\"test_server\", None)\n \n+    def test_recycled_stdio_server_remains_available_for_lazy_reconnect(self):\n+        from tools.mcp_tool import _make_check_fn, _servers\n+\n+        server = _make_mock_server(\"test_server\", session=None)\n+        server._config = {\"command\": \"npx\"}\n+        server._recycled_reason = \"idle_timeout_seconds\"\n+        _servers[\"test_server\"] = server\n+        try:\n+            check = _make_check_fn(\"test_server\")\n+            assert check() is True\n+        finally:\n+            _servers.pop(\"test_server\", None)\n+\n \n # ---------------------------------------------------------------------------\n # MCP loop runner","@@ -742,6 +805,36 @@ def _interrupting_run(coro_or_factory, timeout=30):\n         finally:\n             _servers.pop(\"test_srv\", None)\n \n+    def test_recycled_stdio_server_reconnects_lazily_on_tool_call(self):\n+        from tools.mcp_tool import _make_tool_handler, _servers\n+\n+        mock_session = MagicMock()\n+        mock_session.call_tool = AsyncMock(\n+            return_value=_make_call_result(\"reconnected\", is_error=False)\n+        )\n+        server = _make_mock_server(\"test_srv\", session=None)\n+        server._config = {\"command\": \"npx\"}\n+        server._recycled_reason = \"idle_timeout_seconds\"\n+        _servers[\"test_srv\"] = server\n+\n+        def fake_lazy_reconnect(server_name, srv):\n+            assert server_name == \"test_srv\"\n+            assert srv is server\n+            srv.session = mock_session\n+            srv._recycled_reason = None\n+            return True\n+\n+        try:\n+            handler = _make_tool_handler(\"test_srv\", \"greet\", 120)\n+            with patch(\"tools.mcp_tool._request_lazy_reconnect\", side_effect=fake_lazy_reconnect) as reconnect, \\\n+                 self._patch_mcp_loop():\n+                result = json.loads(handler({\"name\": \"world\"}))\n+            assert result[\"result\"] == \"reconnected\"\n+            reconnect.assert_called_once()\n+            mock_session.call_tool.assert_called_once_with(\"greet\", arguments={\"name\": \"world\"})\n+        finally:\n+            _servers.pop(\"test_srv\", None)\n+\n \n class TestRunOnMCPLoopInterrupts:\n     def test_interrupt_cancels_waiting_mcp_call(self):","@@ -1148,6 +1241,52 @@ async def _test():\n \n         asyncio.run(_test())\n \n+    def test_wait_for_lifecycle_event_recycles_idle_stdio_server(self):\n+        from tools.mcp_tool import MCPServerTask\n+\n+        async def _test():\n+            server = MCPServerTask(\"srv\")\n+            server._config = {\"command\": \"npx\"}\n+            server.session = MagicMock()\n+            server._idle_timeout_seconds = 0.01\n+            server._last_tool_call_at = time.monotonic() - 1.0\n+\n+            reason = await server._wait_for_lifecycle_event()\n+\n+            assert reason == \"recycle\"\n+            assert server._recycled_reason == \"idle_timeout_seconds\"\n+            assert server.session is None\n+\n+        asyncio.run(_test())\n+\n+    def test_stdio_recycle_reason_uses_max_lifetime(self):\n+        from tools.mcp_tool import MCPServerTask\n+\n+        async def _test():\n+            server = MCPServerTask(\"srv\")\n+            server._config = {\"command\": \"npx\"}\n+            server._max_lifetime_seconds = 0.01\n+            server._lifecycle_started_at = time.monotonic() - 1.0\n+\n+            assert server._stdio_recycle_reason() == \"max_lifetime_seconds\"\n+\n+        asyncio.run(_test())\n+\n+    def test_stdio_recycle_deadline_pauses_while_rpc_active(self):\n+        from tools.mcp_tool import MCPServerTask\n+\n+        async def _test():\n+            server = MCPServerTask(\"srv\")\n+            server._config = {\"command\": \"npx\"}\n+            server._idle_timeout_seconds = 0.01\n+            server._last_tool_call_at = time.monotonic() - 1.0\n+\n+            async with server._rpc_lock:\n+                assert server._stdio_recycle_reason() is None\n+                assert server._next_stdio_recycle_deadline() is None\n+\n+        asyncio.run(_test())\n+\n \n # ---------------------------------------------------------------------------\n # discover_mcp_tools toolset injection","@@ -2000,6 +2139,55 @@ async def _test():\n \n         asyncio.run(_test())\n \n+    def test_failed_recycle_reconnect_no_longer_checks_available(self):\n+        \"\"\"A failed lazy reconnect should not leave recycled availability true.\"\"\"\n+        from tools.mcp_tool import (\n+            MCPServerTask,\n+            _MAX_INITIAL_CONNECT_RETRIES,\n+            _make_check_fn,\n+            _servers,\n+        )\n+\n+        run_count = 0\n+        target_server = None\n+\n+        original_run_stdio = MCPServerTask._run_stdio\n+\n+        async def patched_run_stdio(self_srv, config):\n+            nonlocal run_count, target_server\n+            run_count += 1\n+            if target_server is not self_srv:\n+                return await original_run_stdio(self_srv, config)\n+            # After the final retry, run() parks in\n+            # _wait_for_reconnect_or_shutdown(timeout=_PARKED_RETRY_INTERVAL)\n+            # (a real asyncio.wait — the patched asyncio.sleep doesn't cover\n+            # it). Signal shutdown so the park exits immediately instead of\n+            # blocking the test for the 300s self-probe interval.\n+            if run_count > _MAX_INITIAL_CONNECT_RETRIES:\n+                self_srv._shutdown_event.set()\n+            raise ConnectionError(\"recycle reconnect failed\")\n+\n+        async def _test():\n+            nonlocal target_server\n+            server = MCPServerTask(\"test_srv\")\n+            target_server = server\n+            server._config = {\"command\": \"test\"}\n+            server._ready.clear()\n+            server._recycled_reason = \"idle_timeout_seconds\"\n+            _servers[\"test_srv\"] = server\n+            try:\n+                with patch.object(MCPServerTask, \"_run_stdio\", patched_run_stdio), \\\n+                     patch(\"asyncio.sleep\", new_callable=AsyncMock):\n+                    await server.run({\"command\": \"test\"})\n+\n+                assert run_count == _MAX_INITIAL_CONNECT_RETRIES + 1\n+                assert server._recycled_reason is None\n+                assert _make_check_fn(\"test_srv\")() is False\n+            finally:\n+                _servers.pop(\"test_srv\", None)\n+\n+        asyncio.run(_test())\n+\n \n # ---------------------------------------------------------------------------\n # Configurable timeouts"]},{"file":"tests/tools/test_mcp_tool_issue_948.py","hunks":["@@ -1,5 +1,6 @@\n import asyncio\n import os\n+import sys\n from types import SimpleNamespace\n from unittest.mock import AsyncMock, MagicMock, patch\n ","@@ -119,8 +120,18 @@ async def _test():\n             server = MCPServerTask(\"srv\")\n             await server.start({\"command\": \"npx\", \"args\": [\"-y\", \"pkg\"], \"env\": {\"PATH\": \"/usr/bin\"}})\n \n+            # The real (resolved) command no longer reaches StdioServerParameters\n+            # directly -- it's now wrapped in the parent-death watchdog\n+            # supervisor (tools/mcp_stdio_watchdog.py) so an ungraceful exit of\n+            # this process can't orphan it. Assert the resolved npx path and\n+            # its args still flow through correctly as the watchdog's target\n+            # command, preserving this test's original path-resolution intent.\n             call_kwargs = mock_params.call_args.kwargs\n-            assert call_kwargs[\"command\"] == str(npx_path)\n+            assert call_kwargs[\"command\"] == sys.executable\n+            assert call_kwargs[\"args\"][0].endswith(\"mcp_stdio_watchdog.py\")\n+            assert \"--\" in call_kwargs[\"args\"]\n+            sep = call_kwargs[\"args\"].index(\"--\")\n+            assert call_kwargs[\"args\"][sep + 1:] == [str(npx_path), \"-y\", \"pkg\"]\n             assert call_kwargs[\"env\"][\"PATH\"].split(os.pathsep)[0] == str(node_bin)\n \n             await server.shutdown()"]},{"file":"tools/mcp_stdio_watchdog.py","hunks":["@@ -0,0 +1,184 @@\n+#!/usr/bin/env python3\n+\"\"\"Parent-death watchdog supervisor for stdio MCP subprocesses.\n+\n+Problem this fixes (#TBD): a stdio MCP server (e.g. ``npx -y mcp-remote\n+<url>``) is spawned as a direct child of the Hermes process. Hermes's own\n+teardown path (``MCPServerTask.shutdown()`` / ``_kill_orphaned_mcp_children``\n+at final exit) reaps it cleanly on a *graceful* exit. But if the spawning\n+Hermes process dies hard — ``kill -9``, an OS-level crash, a force-quit of\n+the TUI/desktop app — that teardown code never runs, and the child (plus any\n+of its own descendants, e.g. mcp-remote's spawned ``node`` process) is\n+orphaned. macOS has no direct equivalent of Linux's\n+``prctl(PR_SET_PDEATHSIG)`` to make the kernel auto-kill a child when its\n+parent dies, so nothing reaps these until the next Hermes startup's opt-in\n+``_kill_orphaned_mcp_children()`` sweep — which only runs if something calls\n+it. Repeated ungraceful session restarts can pile up N orphaned processes,\n+all racing to hold the same upstream SSE session, producing errors like\n+\"Invalid request parameters\" / \"Received request before initialization was\n+complete\" on the *legitimate* new connection.\n+\n+Fix: don't spawn the MCP server command directly. Spawn this supervisor\n+instead, which:\n+  1. execs the real command as its own child (own process group via\n+     ``start_new_session``, so it doesn't inherit the supervisor's\n+     controlling terminal weirdly and so we can killpg it cleanly);\n+  2. transparently passes stdin/stdout/stderr through — the MCP stdio\n+     protocol talks directly over those pipes, so the supervisor must be a\n+     no-op relay, not a bytes-in-the-middle proxy;\n+  3. runs a background thread that polls the ORIGINAL parent PID using the\n+     exact same orphan-detection algorithm already proven in\n+     ``tui_gateway/slash_worker.py`` (``_is_orphaned``): compare current\n+     ``getppid()`` against the recorded original, and guard PID reuse via\n+     ``psutil`` process creation time;\n+  4. the instant the original parent is gone, terminates the real child's\n+     process group (SIGTERM, grace period, then SIGKILL) and exits.\n+\n+This is intentionally a thin, dependency-light script (``psutil`` only,\n+already a hard dependency via ``tui_gateway/slash_worker.py``) so it starts\n+fast and can't itself become a resource leak.\n+\n+Usage (see ``tools/mcp_tool.py::_run_stdio``)::\n+\n+    python3 -m tools.mcp_stdio_watchdog \\\\\n+        --ppid <original_parent_pid> --create-time <original_parent_create_time> \\\\\n+        -- <real_command> <arg1> <arg2> ...\n+\"\"\"\n+\n+from __future__ import annotations\n+\n+import argparse\n+import os\n+import signal\n+import subprocess\n+import sys\n+import threading\n+import time\n+\n+try:\n+    import psutil\n+except ImportError:  # pragma: no cover - psutil is a hard dependency elsewhere\n+    psutil = None\n+\n+_POLL_INTERVAL_S = 2.0\n+_TERM_GRACE_S = 3.0\n+\n+\n+def _is_orphaned(original_ppid: int, parent_create_time: float, getppid=os.getppid) -> bool:\n+    \"\"\"Mirrors ``tui_gateway.slash_worker._is_orphaned`` exactly.\n+\n+    True once the process that spawned us is gone. Never trusts a bare\n+    ``getppid() == 1`` check (Linux reparents orphans to a subreaper, not\n+    always PID 1), and guards against PID reuse via the recorded creation\n+    time of the original parent.\n+    \"\"\"\n+    if getppid() != original_ppid:\n+        return True\n+    if psutil is None:\n+        # No reliable staleness check available; fall back to the ppid\n+        # comparison alone (still catches the common case).\n+        return False\n+    try:\n+        if not psutil.pid_exists(original_ppid):\n+            return True\n+        return psutil.Process(original_ppid).create_time() != parent_create_time\n+    except psutil.Error:\n+        return True\n+\n+\n+def _terminate_process_group(proc: subprocess.Popen) -> None:\n+    \"\"\"Best-effort SIGTERM-then-SIGKILL of the child's process group.\n+\n+    This module only ever runs on POSIX (the wrap site in tools/mcp_tool.py\n+    gates on ``os.name == \"posix\"``), but guard the POSIX-only primitives\n+    anyway so an accidental Windows import/execute degrades to a plain\n+    child kill instead of AttributeError.\n+    \"\"\"\n+    killpg = getattr(os, \"killpg\", None)\n+    if killpg is None:  # windows-footgun: ok — non-POSIX fallback\n+        try:\n+            proc.terminate()\n+            proc.wait(timeout=_TERM_GRACE_S)\n+        except (OSError, subprocess.TimeoutExpired):\n+            proc.kill()\n+        return\n+    try:\n+        pgid = os.getpgid(proc.pid)\n+    except (ProcessLookupError, OSError):\n+        return\n+    sigkill = getattr(signal, \"SIGKILL\", signal.SIGTERM)\n+    for sig in (signal.SIGTERM, sigkill):\n+        try:\n+            killpg(pgid, sig)\n+        except (ProcessLookupError, PermissionError, OSError):\n+            return\n+        try:\n+            proc.wait(timeout=_TERM_GRACE_S)\n+            return\n+        except subprocess.TimeoutExpired:\n+            continue\n+\n+\n+def _watchdog_loop(proc: subprocess.Popen, original_ppid: int, parent_create_time: float) -> None:\n+    while proc.poll() is None:\n+        if _is_orphaned(original_ppid, parent_create_time):\n+            _terminate_process_group(proc)\n+            return\n+        time.sleep(_POLL_INTERVAL_S)\n+\n+\n+def main(argv: list[str] | None = None) -> int:\n+    parser = argparse.ArgumentParser(\n+        description=\"Parent-death watchdog for a stdio MCP subprocess.\",\n+    )\n+    parser.add_argument(\"--ppid\", type=int, required=True)\n+    parser.add_argument(\"--create-time\", type=float, required=True)\n+    parser.add_argument(\"command\", nargs=argparse.REMAINDER)\n+    args = parser.parse_args(argv)\n+\n+    real_argv = list(args.command)\n+    if real_argv and real_argv[0] == \"--\":\n+        real_argv = real_argv[1:]\n+    if not real_argv:\n+        print(\"mcp_stdio_watchdog: no command given after '--'\", file=sys.stderr)\n+        return 2\n+\n+    # New process group so we can killpg() the whole tree the real command\n+    # may spawn (e.g. mcp-remote's own child `node` process), without\n+    # touching our own group or the (already-gone) original parent's.\n+    proc = subprocess.Popen(\n+        real_argv,\n+        stdin=sys.stdin,\n+        stdout=sys.stdout,\n+        stderr=sys.stderr,\n+        start_new_session=True,\n+    )\n+\n+    # Because the real server lives in its OWN process group (above), the\n+    # parent's graceful-shutdown killpg of *our* group no longer reaches it.\n+    # Forward SIGTERM/SIGINT to the child's group so graceful teardown\n+    # (`_kill_orphaned_mcp_children`, shutdown sweeps) still kills a wedged\n+    # server that ignores stdin EOF — otherwise the watchdog wrap would\n+    # invert the bug it fixes.\n+    def _forward_shutdown(signum, frame):  # noqa: ARG001\n+        _terminate_process_group(proc)\n+        sys.exit(128 + signum)\n+\n+    signal.signal(signal.SIGTERM, _forward_shutdown)\n+    signal.signal(signal.SIGINT, _forward_shutdown)\n+\n+    watchdog = threading.Thread(\n+        target=_watchdog_loop,\n+        args=(proc, args.ppid, args.create_time),\n+        daemon=True,\n+    )\n+    watchdog.start()\n+\n+    try:\n+        return proc.wait()\n+    except KeyboardInterrupt:\n+        _terminate_process_group(proc)\n+        return 130\n+\n+\n+if __name__ == \"__main__\":\n+    sys.exit(main())"]},{"file":"tools/mcp_tool.py","hunks":["@@ -23,6 +23,10 @@\n                                 # 180). Set below the server's session TTL for\n                                 # servers that GC idle sessions quickly (e.g.\n                                 # Unreal Engine editor MCP, ~15s). Floored at 5s.\n+        idle_timeout_seconds: 3600      # optional stdio recycle after idle\n+        max_lifetime_seconds: 86400     # optional stdio recycle after age\n+        # The recycle settings may also live under lifecycle: {...}.\n+        # Use 0 to disable either recycle limit.\n       github:\n         command: \"npx\"\n         args: [\"-y\", \"@modelcontextprotocol/server-github\"]","@@ -330,6 +334,7 @@ def _check_logging_callback_support() -> bool:\n # server is unrevivable: its tools are out of the registry, so no tool call\n # can ever reach the circuit-breaker half-open probe or _signal_reconnect.\n _PARKED_RETRY_INTERVAL = 300     # seconds between parked self-probes\n+_RECYCLED_RECONNECT_TIMEOUT = 15.0\n \n # Keepalive cadence for HTTP/SSE sessions. The MCP spec lets a server expire\n # idle sessions on any TTL it chooses (Streamable HTTP \"Session Management\"),","@@ -616,6 +621,41 @@ def _resolve_stdio_command(command: str, env: dict) -> tuple[str, dict]:\n     return resolved_command, resolved_env\n \n \n+def _wrap_command_with_watchdog(command: str, args: list) -> tuple[str, list]:\n+    \"\"\"Wrap a stdio MCP server command in the parent-death watchdog supervisor.\n+\n+    See ``tools/mcp_stdio_watchdog.py`` module docstring for the full\n+    rationale. Returns the (command, args) unchanged on any platform/failure\n+    where the wrap can't safely apply, so this can never be the reason a\n+    previously-working MCP server stops starting.\n+    \"\"\"\n+    if os.name != \"posix\":\n+        # Relies on process groups (os.getpgid/os.killpg); no POSIX\n+        # equivalent wired up here yet, matching the existing killpg-based\n+        # orphan cleanup's platform scope (Windows falls back to plain\n+        # os.kill there too).\n+        return command, args\n+    try:\n+        my_pid = os.getpid()\n+        try:\n+            import psutil\n+            create_time = psutil.Process(my_pid).create_time()\n+        except ImportError:\n+            create_time = time.time()\n+    except Exception:\n+        # Never let watchdog bookkeeping failure block a real MCP connection.\n+        return command, args\n+    watchdog_args = [\n+        os.path.join(os.path.dirname(os.path.abspath(__file__)), \"mcp_stdio_watchdog.py\"),\n+        \"--ppid\", str(my_pid),\n+        \"--create-time\", repr(create_time),\n+        \"--\",\n+        command,\n+        *args,\n+    ]\n+    return sys.executable, watchdog_args\n+\n+\n # ---------------------------------------------------------------------------\n # MCP ImageContent block → Hermes MEDIA tag\n # ---------------------------------------------------------------------------","@@ -1483,6 +1523,8 @@ class MCPServerTask:\n         \"_registered_tool_names\", \"_auth_type\", \"_refresh_lock\",\n         \"_rpc_lock\", \"_pending_refresh_tasks\",\n         \"_pending_call_context\",\n+        \"_lifecycle_started_at\", \"_last_tool_call_at\",\n+        \"_idle_timeout_seconds\", \"_max_lifetime_seconds\", \"_recycled_reason\",\n         \"initialize_result\", \"_ping_unsupported\",\n         \"_reconnect_retries\",\n     )","@@ -1527,6 +1569,12 @@ def __init__(self, name: str):\n         # gateway-platform attribution and routes the approval prompt\n         # to the right surface (Telegram, Slack, etc.).\n         self._pending_call_context: Optional[contextvars.Context] = None\n+        now = time.monotonic()\n+        self._lifecycle_started_at: float = now\n+        self._last_tool_call_at: float = now\n+        self._idle_timeout_seconds: Optional[float] = None\n+        self._max_lifetime_seconds: Optional[float] = None\n+        self._recycled_reason: Optional[str] = None\n         # Captures the ``InitializeResult`` returned by\n         # ``await session.initialize()`` so downstream code can inspect the\n         # server's real advertised capabilities (``.capabilities.resources``,","@@ -1564,6 +1612,53 @@ def _advertises_tools(self) -> bool:\n             return True\n         return getattr(caps, \"tools\", None) is not None\n \n+    def _is_recycled_stdio(self) -> bool:\n+        \"\"\"Return True when a stdio server was intentionally recycled.\"\"\"\n+        return not self._is_http() and self._recycled_reason is not None\n+\n+    def mark_tool_call(self) -> None:\n+        \"\"\"Record that a user-visible MCP operation is starting.\"\"\"\n+        self._last_tool_call_at = time.monotonic()\n+\n+    def _mark_lifecycle_started(self) -> None:\n+        now = time.monotonic()\n+        self._lifecycle_started_at = now\n+        self._last_tool_call_at = now\n+        self._recycled_reason = None\n+\n+    def _stdio_recycle_reason(self, now: Optional[float] = None) -> Optional[str]:\n+        \"\"\"Return the stdio recycle reason if idle/age limits have elapsed.\"\"\"\n+        if self._is_http() or self._rpc_lock.locked():\n+            return None\n+        now = time.monotonic() if now is None else now\n+        if (\n+            self._max_lifetime_seconds is not None\n+            and now - self._lifecycle_started_at >= self._max_lifetime_seconds\n+        ):\n+            return \"max_lifetime_seconds\"\n+        if (\n+            self._idle_timeout_seconds is not None\n+            and now - self._last_tool_call_at >= self._idle_timeout_seconds\n+        ):\n+            return \"idle_timeout_seconds\"\n+        return None\n+\n+    def _next_stdio_recycle_deadline(self) -> Optional[float]:\n+        \"\"\"Return the next monotonic recycle deadline for stdio, if any.\"\"\"\n+        if self._is_http() or self._rpc_lock.locked():\n+            return None\n+        deadlines = []\n+        if self._max_lifetime_seconds is not None:\n+            deadlines.append(self._lifecycle_started_at + self._max_lifetime_seconds)\n+        if self._idle_timeout_seconds is not None:\n+            deadlines.append(self._last_tool_call_at + self._idle_timeout_seconds)\n+        return min(deadlines) if deadlines else None\n+\n+    def _mark_stdio_recycled(self, reason: str) -> None:\n+        \"\"\"Mark a stdio session dormant before its transport finishes closing.\"\"\"\n+        self._recycled_reason = reason\n+        self.session = None\n+\n     # ----- Dynamic tool discovery (notifications/tools/list_changed) -----\n \n     async def _refresh_tools_task(self):","@@ -1776,6 +1871,9 @@ async def _wait_for_lifecycle_event(self) -> str:\n                         tokens, new session ID, etc.). The reconnect event\n                         is cleared before return so the next cycle starts\n                         with a fresh signal.\n+            \"recycle\"   if a stdio idle/max-lifetime limit elapsed. The\n+                        current transport is torn down and restarted lazily\n+                        on the next tool call.\n \n         Shutdown takes precedence if both events are set simultaneously.\n ","@@ -1807,14 +1905,29 @@ async def _wait_for_lifecycle_event(self) -> str:\n         reconnect_task = asyncio.create_task(self._reconnect_event.wait())\n         try:\n             while True:\n+                recycle_reason = self._stdio_recycle_reason()\n+                if recycle_reason is not None:\n+                    self._mark_stdio_recycled(recycle_reason)\n+                    return \"recycle\"\n+\n+                timeout = keepalive_interval\n+                recycle_deadline = self._next_stdio_recycle_deadline()\n+                if recycle_deadline is not None:\n+                    timeout = max(0.0, min(timeout, recycle_deadline - time.monotonic()))\n+\n                 done, _pending = await asyncio.wait(\n                     {shutdown_task, reconnect_task},\n-                    timeout=keepalive_interval,\n+                    timeout=timeout,\n                     return_when=asyncio.FIRST_COMPLETED,\n                 )\n                 if done:\n                     break\n \n+                recycle_reason = self._stdio_recycle_reason()\n+                if recycle_reason is not None:\n+                    self._mark_stdio_recycled(recycle_reason)\n+                    return \"recycle\"\n+\n                 # Timeout — no lifecycle event fired.  Probe the connection\n                 # to detect stale/expired sessions. Prefer ``ping`` (MCP base\n                 # protocol liveness): it works uniformly and stays a few bytes","@@ -1919,6 +2032,9 @@ async def _run_stdio(self, config: dict):\n         # it with a wall-clock timeout so a stalled SSL handshake can't freeze\n         # MCP discovery / gateway startup (#29184). The check is fail-open, so\n         # on timeout we log and proceed rather than blocking indefinitely.\n+        # NOTE: must run against the REAL command/args — the watchdog wrap\n+        # below rewrites argv to `python -m tools.mcp_stdio_watchdog …`,\n+        # which would silently turn the preflight into a no-op.\n         from tools.osv_check import check_package_for_malware\n         try:\n             malware_error = await asyncio.wait_for(","@@ -1937,6 +2053,18 @@ async def _run_stdio(self, config: dict):\n                 f\"MCP server '{self.name}': {malware_error}\"\n             )\n \n+        # Wrap the real command in a parent-death watchdog supervisor so an\n+        # ungraceful exit of this Hermes process (kill -9, crash, force-quit)\n+        # can't leave the stdio MCP child (and its own descendants, e.g.\n+        # mcp-remote's spawned `node`) running forever. On a clean exit,\n+        # MCPServerTask.shutdown() / _kill_orphaned_mcp_children() still do\n+        # the reaping as before -- this only covers the case where that code\n+        # never gets to run. POSIX-only (relies on process groups); no-op\n+        # elsewhere, matching existing killpg-based cleanup's platform scope.\n+        # Applied AFTER the OSV preflight so the check inspects the real\n+        # package, not the watchdog wrapper.\n+        command, args = _wrap_command_with_watchdog(command, args)\n+\n         server_params = StdioServerParameters(\n             command=command,\n             args=args,","@@ -1951,6 +2079,18 @@ async def _run_stdio(self, config: dict):\n         if _MCP_LOGGING_CALLBACK_SUPPORTED:\n             sampling_kwargs[\"logging_callback\"] = self._make_logging_callback()\n \n+        # Reap any orphaned subprocesses from prior failed connection\n+        # attempts before spawning a new one.  Without this, each retry in\n+        # the run() reconnect loop spawns a fresh process pair while the\n+        # previous failed pair lingers — leading to rapid zombie\n+        # accumulation (see #57355, #57228).  The unscoped sweep also\n+        # opportunistically reaps orphans left by *other* servers that\n+        # never reconnect; per-server filtering via ``server_name`` remains\n+        # available for scoped call sites.  Run in a worker thread: the\n+        # reaper blocks up to 2s (SIGTERM → wait → SIGKILL) when orphans\n+        # exist, which would otherwise stall the shared MCP event loop.\n+        await asyncio.to_thread(_kill_orphaned_mcp_children)\n+\n         # Snapshot child PIDs before spawning so we can track the new one.\n         pids_before = _snapshot_child_pids()\n         new_pids: set = set()","@@ -1995,8 +2135,25 @@ async def _run_stdio(self, config: dict):\n                 async with ClientSession(\n                     read_stream, write_stream, **sampling_kwargs\n                 ) as session:\n-                    self.initialize_result = await session.initialize()\n+                    # Bound the MCP handshake. A stdio server that never\n+                    # completes ``initialize`` (e.g. emits a non-JSON-RPC frame\n+                    # and then blocks on stdin) otherwise hangs this coroutine\n+                    # forever on the background loop: ``connect_timeout`` only\n+                    # bounds the caller's ``.result()`` wait, not the coroutine\n+                    # itself. Because the connect never unwinds, the cleanup\n+                    # ``finally`` below never runs, so the spawned child and its\n+                    # stdio pipes/pidfd leak on every discovery retry — unbounded\n+                    # until the gateway hits EMFILE. Timing out here converts the\n+                    # hang into a normal failure, letting the ``finally`` reap the\n+                    # child. See #59349.\n+                    connect_timeout = float(\n+                        config.get(\"connect_timeout\", _DEFAULT_CONNECT_TIMEOUT)\n+                    )\n+                    self.initialize_result = await asyncio.wait_for(\n+                        session.initialize(), timeout=connect_timeout\n+                    )\n                     self.session = session\n+                    self._mark_lifecycle_started()\n                     await self._discover_tools()\n                     self._ready.set()\n                     # Session is live again: clear any breaker state from a","@@ -2010,7 +2167,7 @@ async def _run_stdio(self, config: dict):\n                     # stdio transport does not use OAuth, but we still honor\n                     # _reconnect_event (e.g. future manual /mcp refresh) for\n                     # consistency with _run_http.\n-                    await self._wait_for_lifecycle_event()\n+                    return await self._wait_for_lifecycle_event()\n         finally:\n             # Runs on clean exit, exceptions, AND asyncio cancellation.\n             # If any of the spawned PIDs are still alive, the SDK's","@@ -2041,6 +2198,7 @@ async def _run_stdio(self, config: dict):\n                                 pgroup_alive = False\n                         if pid_alive or pgroup_alive:\n                             _orphan_stdio_pids.add(pid)\n+                            _orphan_stdio_pid_servers[pid] = self.name\n                         else:\n                             # Nothing left to reap — drop the pgid entry so\n                             # PID-reuse can't surface stale pgroup state later.","@@ -2283,7 +2441,13 @@ def _mcp_http_client_factory(\n                 async with ClientSession(\n                     read_stream, write_stream, **sampling_kwargs\n                 ) as session:\n-                    self.initialize_result = await session.initialize()\n+                    # Bound the handshake — same orphaned-task hang as the\n+                    # stdio path (#59349): an endpoint that accepts the\n+                    # connection but never answers ``initialize`` parks this\n+                    # coroutine forever on the background loop.\n+                    self.initialize_result = await asyncio.wait_for(\n+                        session.initialize(), timeout=float(connect_timeout)\n+                    )\n                     self.session = session\n                     await self._discover_tools()\n                     self._ready.set()","@@ -2298,7 +2462,7 @@ def _mcp_http_client_factory(\n                             \"MCP server '%s': reconnect requested — \"\n                             \"tearing down SSE session\", self.name,\n                         )\n-            return\n+            return reason\n \n         if _MCP_NEW_HTTP:\n             # New API (mcp >= 1.24.0): build an explicit httpx.AsyncClient","@@ -2337,7 +2501,10 @@ async def _strip_auth_on_cross_origin_redirect(response):\n                     read_stream, write_stream, _get_session_id,\n                 ):\n                     async with ClientSession(read_stream, write_stream, **sampling_kwargs) as session:\n-                        self.initialize_result = await session.initialize()\n+                        # Bound the handshake (#59349) — see stdio path.\n+                        self.initialize_result = await asyncio.wait_for(\n+                            session.initialize(), timeout=float(connect_timeout)\n+                        )\n                         self.session = session\n                         await self._discover_tools()\n                         self._ready.set()","@@ -2352,6 +2519,7 @@ async def _strip_auth_on_cross_origin_redirect(response):\n                                 \"MCP server '%s': reconnect requested — \"\n                                 \"tearing down HTTP session\", self.name,\n                             )\n+            return reason\n         else:\n             # Deprecated API (mcp < 1.24.0): manages httpx client internally.\n             _http_kwargs: dict = {","@@ -2365,7 +2533,10 @@ async def _strip_auth_on_cross_origin_redirect(response):\n                 read_stream, write_stream, _get_session_id,\n             ):\n                 async with ClientSession(read_stream, write_stream, **sampling_kwargs) as session:\n-                    self.initialize_result = await session.initialize()\n+                    # Bound the handshake (#59349) — see stdio path.\n+                    self.initialize_result = await asyncio.wait_for(\n+                        session.initialize(), timeout=float(connect_timeout)\n+                    )\n                     self.session = session\n                     await self._discover_tools()\n                     self._ready.set()","@@ -2380,6 +2551,7 @@ async def _strip_auth_on_cross_origin_redirect(response):\n                             \"MCP server '%s': reconnect requested — \"\n                             \"tearing down legacy HTTP session\", self.name,\n                         )\n+            return reason\n \n     async def _discover_tools(self):\n         \"\"\"Discover tools from the connected session.","@@ -2440,6 +2612,8 @@ async def run(self, config: dict):\n         self._config = config\n         self.tool_timeout = config.get(\"timeout\", _DEFAULT_TOOL_TIMEOUT)\n         self._auth_type = (config.get(\"auth\") or \"\").lower().strip()\n+        self._idle_timeout_seconds = _get_lifecycle_seconds(config, \"idle_timeout_seconds\")\n+        self._max_lifetime_seconds = _get_lifecycle_seconds(config, \"max_lifetime_seconds\")\n \n         # Set up sampling handler if enabled and SDK types are available\n         sampling_config = config.get(\"sampling\", {})","@@ -2515,9 +2689,9 @@ async def run(self, config: dict):\n         while True:\n             try:\n                 if self._is_http():\n-                    await self._run_http(config)\n+                    lifecycle_reason = await self._run_http(config)\n                 else:\n-                    await self._run_stdio(config)\n+                    lifecycle_reason = await self._run_stdio(config)\n                 # Transport returned cleanly. Two cases:\n                 #  - _shutdown_event was set: exit the run loop entirely.\n                 #  - _reconnect_event was set (auth recovery): loop back and","@@ -2525,6 +2699,18 @@ async def run(self, config: dict):\n                 #    touch the retry counters — this is not a failure.\n                 if self._shutdown_event.is_set():\n                     break\n+                if lifecycle_reason == \"recycle\":\n+                    logger.info(\n+                        \"MCP server '%s': stdio session recycled after %s; \"\n+                        \"waiting for lazy reconnect\",\n+                        self.name, self._recycled_reason,\n+                    )\n+                    self.session = None\n+                    await self._wait_for_lazy_reconnect()\n+                    if self._shutdown_event.is_set():\n+                        break\n+                    self._reconnect_event.clear()\n+                    continue\n                 logger.info(\n                     \"MCP server '%s': reconnecting (OAuth recovery or \"\n                     \"manual refresh)\",","@@ -2560,6 +2746,13 @@ async def run(self, config: dict):\n                 raise\n             except Exception as exc:\n                 self.session = None\n+                if self._is_recycled_stdio():\n+                    logger.warning(\n+                        \"MCP server '%s': lazy reconnect after stdio recycle \"\n+                        \"failed, marking unavailable while retrying: %s\",\n+                        self.name, exc,\n+                    )\n+                    self._recycled_reason = None\n \n                 # If this is the first connection attempt, retry with backoff\n                 # before giving up. A transient DNS/network blip at startup","@@ -2686,7 +2879,19 @@ async def run(self, config: dict):\n     async def start(self, config: dict):\n         \"\"\"Create the background Task and wait until ready (or failed).\"\"\"\n         self._task = asyncio.ensure_future(self.run(config))\n-        await self._ready.wait()\n+        try:\n+            await self._ready.wait()\n+        except asyncio.CancelledError:\n+            # The caller's connect timeout (discover_mcp_tools wraps start()\n+            # in asyncio.wait_for) cancels *this* coroutine, but the\n+            # ensure_future'd run() task is independent and would otherwise\n+            # keep running detached — parked on a hung transport with no\n+            # owner to reap it (#59349). Propagate the cancellation so the\n+            # transport context managers unwind and their finally blocks\n+            # release the child process / FDs.\n+            if self._task and not self._task.done():\n+                self._task.cancel()\n+            raise\n         if self._error:\n             raise self._error\n ","@@ -2736,6 +2941,24 @@ def _deregister_tools(self) -> None:\n             _forget_mcp_tool_server(tool_name)\n         self._registered_tool_names = []\n \n+    async def _wait_for_lazy_reconnect(self) -> None:\n+        \"\"\"Wait while an intentionally recycled stdio server is dormant.\"\"\"\n+        shutdown_task = asyncio.create_task(self._shutdown_event.wait())\n+        reconnect_task = asyncio.create_task(self._reconnect_event.wait())\n+        try:\n+            await asyncio.wait(\n+                {shutdown_task, reconnect_task},\n+                return_when=asyncio.FIRST_COMPLETED,\n+            )\n+        finally:\n+            for task in (shutdown_task, reconnect_task):\n+                if not task.done():\n+                    task.cancel()\n+                    try:\n+                        await task\n+                    except (asyncio.CancelledError, Exception):\n+                        pass\n+\n \n # ---------------------------------------------------------------------------\n # Module-level state","@@ -3234,6 +3457,7 @@ def _handle_session_expired_and_retry(\n # Separate from _stdio_pids so cleanup sweeps never race with active\n # sessions (e.g. concurrent cron jobs or live user chats).\n _orphan_stdio_pids: set = set()\n+_orphan_stdio_pid_servers: Dict[int, str] = {}\n \n # Process-group IDs of stdio MCP subprocesses, captured at spawn time.\n # The MCP SDK spawns stdio children with ``start_new_session=True`` so each","@@ -3580,6 +3804,58 @@ async def _connect_server(name: str, config: dict) -> MCPServerTask:\n # Handler / check-fn factories\n # ---------------------------------------------------------------------------\n \n+def _request_lazy_reconnect(server_name: str, server: MCPServerTask) -> bool:\n+    \"\"\"Wake a recycled stdio server and wait briefly for a fresh session.\"\"\"\n+    if not server._is_recycled_stdio():\n+        return False\n+\n+    with _lock:\n+        loop = _mcp_loop\n+    if loop is None or not loop.is_running():\n+        return False\n+\n+    def _signal_reconnect() -> None:\n+        server._ready.clear()\n+        server._reconnect_event.set()\n+\n+    loop.call_soon_threadsafe(_signal_reconnect)\n+\n+    async def _await_ready() -> bool:\n+        deadline = time.monotonic() + _RECYCLED_RECONNECT_TIMEOUT\n+        while time.monotonic() < deadline:\n+            if server.session is not None and server._ready.is_set():\n+                return True\n+            await asyncio.sleep(0.05)\n+        return False\n+\n+    try:\n+        return bool(_run_on_mcp_loop(_await_ready, timeout=_RECYCLED_RECONNECT_TIMEOUT))\n+    except Exception as exc:\n+        logger.warning(\n+            \"MCP server '%s': lazy reconnect after stdio recycle failed: %s\",\n+            server_name, exc,\n+        )\n+        return False\n+\n+\n+def _get_connected_server_for_call(server_name: str) -> Optional[MCPServerTask]:\n+    \"\"\"Return a connected server, lazily reconnecting recycled stdio state.\"\"\"\n+    with _lock:\n+        server = _servers.get(server_name)\n+    if server is not None and server.session is None and server._is_recycled_stdio():\n+        _request_lazy_reconnect(server_name, server)\n+        with _lock:\n+            server = _servers.get(server_name)\n+    return server\n+\n+\n+def _mark_server_call_started(server: Any) -> None:\n+    \"\"\"Record a user-visible MCP operation when the server supports it.\"\"\"\n+    mark_tool_call = getattr(server, \"mark_tool_call\", None)\n+    if callable(mark_tool_call):\n+        mark_tool_call()\n+\n+\n def _make_tool_handler(server_name: str, tool_name: str, tool_timeout: float):\n     \"\"\"Return a sync handler that calls an MCP tool via the background loop.\n ","@@ -3614,8 +3890,7 @@ def _handler(args: dict, **kwargs) -> str:\n                 }, ensure_ascii=False)\n             # Cooldown elapsed → fall through as a half-open probe.\n \n-        with _lock:\n-            server = _servers.get(server_name)\n+        server = _get_connected_server_for_call(server_name)\n         if not server:\n             _bump_server_error(server_name)\n             return json.dumps({","@@ -3657,6 +3932,7 @@ def _handler(args: dict, **kwargs) -> str:\n                 }, ensure_ascii=False)\n \n         async def _call():\n+            _mark_server_call_started(server)\n             async with server._rpc_lock:\n                 # Snapshot the agent's context so an elicitation callback\n                 # triggered during this call (fired on the MCP recv loop","@@ -3770,14 +4046,14 @@ def _make_list_resources_handler(server_name: str, tool_timeout: float):\n     \"\"\"Return a sync handler that lists resources from an MCP server.\"\"\"\n \n     def _handler(args: dict, **kwargs) -> str:\n-        with _lock:\n-            server = _servers.get(server_name)\n+        server = _get_connected_server_for_call(server_name)\n         if not server or not server.session:\n             return json.dumps({\n                 \"error\": f\"MCP server '{server_name}' is not connected\"\n             }, ensure_ascii=False)\n \n         async def _call():\n+            _mark_server_call_started(server)\n             async with server._rpc_lock:\n                 result = await server.session.list_resources()\n             resources = []","@@ -3830,8 +4106,7 @@ def _make_read_resource_handler(server_name: str, tool_timeout: float):\n     def _handler(args: dict, **kwargs) -> str:\n         from tools.registry import tool_error\n \n-        with _lock:\n-            server = _servers.get(server_name)\n+        server = _get_connected_server_for_call(server_name)\n         if not server or not server.session:\n             return json.dumps({\n                 \"error\": f\"MCP server '{server_name}' is not connected\"","@@ -3842,6 +4117,7 @@ def _handler(args: dict, **kwargs) -> str:\n             return tool_error(\"Missing required parameter 'uri'\")\n \n         async def _call():\n+            _mark_server_call_started(server)\n             async with server._rpc_lock:\n                 result = await server.session.read_resource(uri)\n             # read_resource returns ReadResourceResult with .contents list","@@ -3888,14 +4164,14 @@ def _make_list_prompts_handler(server_name: str, tool_timeout: float):\n     \"\"\"Return a sync handler that lists prompts from an MCP server.\"\"\"\n \n     def _handler(args: dict, **kwargs) -> str:\n-        with _lock:\n-            server = _servers.get(server_name)\n+        server = _get_connected_server_for_call(server_name)\n         if not server or not server.session:\n             return json.dumps({\n                 \"error\": f\"MCP server '{server_name}' is not connected\"\n             }, ensure_ascii=False)\n \n         async def _call():\n+            _mark_server_call_started(server)\n             async with server._rpc_lock:\n                 result = await server.session.list_prompts()\n             prompts = []","@@ -3953,8 +4229,7 @@ def _make_get_prompt_handler(server_name: str, tool_timeout: float):\n     def _handler(args: dict, **kwargs) -> str:\n         from tools.registry import tool_error\n \n-        with _lock:\n-            server = _servers.get(server_name)\n+        server = _get_connected_server_for_call(server_name)\n         if not server or not server.session:\n             return json.dumps({\n                 \"error\": f\"MCP server '{server_name}' is not connected\"","@@ -3966,6 +4241,7 @@ def _handler(args: dict, **kwargs) -> str:\n         arguments = args.get(\"arguments\", {})\n \n         async def _call():\n+            _mark_server_call_started(server)\n             async with server._rpc_lock:\n                 result = await server.session.get_prompt(name, arguments=arguments)\n             # GetPromptResult has .messages list","@@ -4024,7 +4300,10 @@ def _make_check_fn(server_name: str):\n     def _check() -> bool:\n         with _lock:\n             server = _servers.get(server_name)\n-        return server is not None and server.session is not None\n+        return (\n+            server is not None\n+            and (server.session is not None or server._is_recycled_stdio())\n+        )\n \n     return _check\n ","@@ -4324,6 +4603,27 @@ def _parse_boolish(value: Any, default: bool = True) -> bool:\n     return default\n \n \n+def _get_lifecycle_seconds(config: dict, key: str) -> Optional[float]:\n+    \"\"\"Return an optional positive lifecycle timeout from top-level/nested config.\"\"\"\n+    raw = config.get(key)\n+    lifecycle = config.get(\"lifecycle\")\n+    if raw is None and isinstance(lifecycle, dict):\n+        raw = lifecycle.get(key)\n+    if raw is None:\n+        return None\n+    try:\n+        seconds = float(raw)\n+    except (TypeError, ValueError):\n+        logger.warning(\"MCP config %s must be a number of seconds; ignoring %r\", key, raw)\n+        return None\n+    if seconds == 0:\n+        return None\n+    if seconds < 0:\n+        logger.warning(\"MCP config %s must be positive; ignoring %r\", key, raw)\n+        return None\n+    return seconds\n+\n+\n _UTILITY_CAPABILITY_METHODS = {\n     \"list_resources\": \"list_resources\",\n     \"read_resource\": \"read_resource\",","@@ -5161,7 +5461,10 @@ async def _shutdown():\n     _stop_mcp_loop()\n \n \n-def _kill_orphaned_mcp_children(include_active: bool = False) -> None:\n+def _kill_orphaned_mcp_children(\n+    include_active: bool = False,\n+    server_name: Optional[str] = None,\n+) -> None:\n     \"\"\"Best-effort graceful shutdown of stdio MCP subprocesses to reap orphans.\n \n     Orphans are PIDs that survived their session context exit (SDK teardown","@@ -5180,6 +5483,10 @@ def _kill_orphaned_mcp_children(include_active: bool = False) -> None:\n     first) are reaped alongside the direct child.  Falls back to ``os.kill``\n     on Windows and when no pgid is recorded.\n \n+    When ``server_name`` is set, only orphaned PIDs known to belong to that\n+    MCP server are reaped. This lets stdio reconnects clean up their previous\n+    transport without touching unrelated servers.\n+\n     With ``include_active=True`` also kills every PID in ``_stdio_pids`` —\n     used only at final shutdown, after the MCP event loop has stopped and no\n     sessions can still be in flight.","@@ -5189,11 +5496,24 @@ def _kill_orphaned_mcp_children(include_active: bool = False) -> None:\n     with _lock:\n         pids: Dict[int, str] = {}\n         for opid in _orphan_stdio_pids:\n-            pids[opid] = \"orphan\"\n-        _orphan_stdio_pids.clear()\n+            owner = _orphan_stdio_pid_servers.get(opid, \"orphan\")\n+            if server_name is not None and owner != server_name:\n+                continue\n+            pids[opid] = owner\n+        for opid in pids:\n+            _orphan_stdio_pids.discard(opid)\n+            _orphan_stdio_pid_servers.pop(opid, None)\n         if include_active:\n-            pids.update(dict(_stdio_pids))\n-            _stdio_pids.clear()\n+            active = dict(_stdio_pids)\n+            if server_name is not None:\n+                active = {\n+                    pid: owner\n+                    for pid, owner in active.items()\n+                    if owner == server_name\n+                }\n+            pids.update(active)\n+            for pid in active:\n+                _stdio_pids.pop(pid, None)\n         # Snapshot pgids for the pids we're about to kill, then drop the\n         # entries so a future spawn can't collide with stale state.\n         pgids: Dict[int, int] = {pid: _stdio_pgids[pid] for pid in pids if pid in _stdio_pgids}"]},{"file":"website/docs/user-guide/features/mcp.md","hunks":["@@ -296,7 +296,9 @@ Hermes reads MCP config from `~/.hermes/config.yaml` under `mcp_servers`.\n | `client_cert` | string \\| list | Client certificate for mTLS — a combined PEM path, or `[cert, key]` / `[cert, key, password]` |\n | `client_key` | string | Client private-key PEM path (when separate from `client_cert`) |\n | `timeout` | number | Tool call timeout |\n-| `connect_timeout` | number | Initial connection timeout |\n+| `connect_timeout` | number | Initial connection timeout (also bounds the MCP `initialize` handshake) |\n+| `idle_timeout_seconds` | number | Recycle a stdio server after this many seconds without a tool call (`0` = never, default). The server restarts transparently on the next tool call. |\n+| `max_lifetime_seconds` | number | Recycle a stdio server after this total age (`0` = never, default). Restarts transparently on next use. |\n | `enabled` | bool | If `false`, Hermes skips the server entirely |\n | `supports_parallel_tool_calls` | bool | If `true`, tools from this server may run concurrently |\n | `tools` | mapping | Per-server tool filtering and utility policy |","@@ -310,6 +312,23 @@ mcp_servers:\n     args: [\"-y\", \"@modelcontextprotocol/server-filesystem\", \"/tmp\"]\n ```\n \n+### Recycling memory-heavy stdio servers\n+\n+Browser-based MCP servers (e.g. `@playwright/mcp`) keep a full Chromium\n+resident after their first tool call — hundreds of MB that never get\n+released. Opt in to automatic recycling and the server is torn down after\n+the idle/lifetime limit, then restarted transparently the next time one of\n+its tools is called (its tools stay registered the whole time):\n+\n+```yaml\n+mcp_servers:\n+  playwright:\n+    command: \"npx\"\n+    args: [\"-y\", \"@playwright/mcp@latest\", \"--headless\"]\n+    idle_timeout_seconds: 900     # recycle after 15 min without a tool call\n+    max_lifetime_seconds: 86400   # and at least once a day regardless\n+```\n+\n ### Minimal HTTP example\n \n ```yaml\n"]}]}],"trust":{"default_tier":"verified","includes_suggested":false}}