# NousResearch/hermes-agent · Gateway agent cache and resource lifecycle leaks

## Open now

- 2 issues, 1 PR open on GitHub.
- #43657: aiohttp ClientSession leak after auxiliary tasks (title_generation, etc.)
- #64278: \[Bug\]: Gateway soft agent-cache eviction leaves memory providers running and leaks threads/HTTP clients
- PR #64284: fix: clean up evicted memory providers (salvage of #63703 + review fixes)

A cluster of bug reports and fixes centered on memory and resource leaks in the gateway's agent cache, covering missing cleanup on eviction paths, unclosed aiohttp client sessions, and unreleased memory providers from temporary and cached AIAgent instances.

## TL;DR

- 21 member(s): 2 open issue(s), 6 closed issue(s), 1 open PR(s), 4 merged PR(s), 8 PR(s) closed without merging.
- A fix was merged to main in PR #11028. Verify it addresses your case.
- **Recommended:** read PR #11028 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-11028,pr-41975,pr-122925`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text.
- Members outside that structural core may belong to a different underlying problem.
- Pairwise link density `0.08`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Open related issues

- #43657: aiohttp ClientSession leak after auxiliary tasks (title_generation, etc.)
- #64278: \[Bug\]: Gateway soft agent-cache eviction leaves memory providers running and leaks threads/HTTP clients

## Closed / duplicate issues

- #10865: Multiple AIAgent instances leak aiohttp client sessions
- #19251: Investigate residual gateway memory leak under heavy scheduled workload
- #25315: fix: gateway memory leak — _evict_cached_agent drops agents without cleanup, _session_messages never cleared
- #48287: \[Bug\]: Agent cache (_agent_cache) causes 50GB+ virtual memory leak in long-running gateway sessions
- #50197: Multiple AIAgent instances lack lifecycle cleanup (shutdown_memory_provider + close)
- #58817: \[Bug\]: Gateway memory grows ~5x over 20h on Linux — planned-restart masking leak, not fixing it

## Prior & competing PRs

- **#11028: fix(gateway): close temporary agents after one-off tasks**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/run.py`, `tests/gateway/test_background_command.py`, `tests/gateway/test_compress_command.py`, `tests/gateway/test_flush_memory_stale_guard.py`, `tests/gateway/test_session_hygiene.py`
- **#41974: fix(gateway): release evicted agent clients to stop RSS leak (#29298)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/run.py`
  - fix hunks:
  - `gateway/run.py`

    ```diff
    @@ -12493,25 +12493,67 @@ async def _interrupt_and_clear_session(
                 self._release_running_agent_state(session_key)
     
         def _evict_cached_agent(self, session_key: str) -> None:
    -        """Remove a cached agent for a session (called on /new, /model, etc)."""
    +        """Remove a cached agent for a session (called on /new, /model, etc).
    +
    +        Pops the entry AND soft-releases the evicted agent's LLM client
    +        pool so the httpx connection (sockets + held buffers) is freed
    +        promptly rather than waiting on CPython GC — AIAgent holds
    +        reference cycles (callbacks, tool state) that delay refcount
    +        collection, so a manual release is required to keep gateway RSS
    +        flat across many /new, /model, undo and reset operations (#29298,
    +        same leak class as #25315).
    +
    +        The release is soft (``release_clients()``): it frees the client
    +        pool and per-turn child subagents but PRESERVES the session's
    +        terminal sandbox, browser daemon, and tracked bg processes (keyed
    +        on task_id), because the session may resume with a freshly-built
    +        agent.  Call sites that want a hard teardown (true conversation
    +        boundaries like /new) already call ``_cleanup_agent_resources``
    +        before evicting; ``release_clients`` is idempotent and safe to
    +        run again after that (the client is already None).
    +
    +        Cleanup runs on a daemon thread so we never block holding
    +        ``_agent_cache_lock`` on slow socket teardown — mirrors the
    +        cap-enforcer and idle-sweeper paths.
    +        """
             _lock = getattr(self, "_agent_cache_lock", None)
    +        evicted = None
             if _lock:
                 with _lock:
    -                entry = self._agent_cache.pop(session_key, None)
    -            # Release clients on a daemon thread, same as _enforce_agent_cache_cap.
    -            # Without this, every /new, /model, /reasoning, codex-runtime change,
    -            # and /undo leaked a full agent: OpenAI client, httpx transport, SSL
    -            # context, and conversation history. Only the /new path cleaned up
    -            # first; the rest popped the entry and dropped it on the floor.
    -            if entry is not None:
    -                agent = entry[0] if isinstance(entry, tuple) and entry else None
    -                if agent is not None:
    -                    threading.Thread(
    -                        target=self._release_evicted_agent_soft,
    -                        args=(agent,),
    -                        daemon=True,
    -                        name=f"agent-cache-cmd-evict-{session_key[:24]}",
    -                    ).start()
    +                evicted = self._agent_cache.pop(session_key, None)
    +        else:
    +            _cache = getattr(self, "_agent_cache", None)
    +            if _cache is not None:
    +                evicted = _cache.pop(session_key, None)
    +
    +        agent = evicted[0] if isinstance(evicted, tuple) and evicted else evicted
    +        if agent is None or agent is _AGENT_PENDING_SENTINEL:
    +            return
    +
    +        # Don't tear down an agent that's actively mid-turn — its client,
    +        # sandbox and child subagents are in use by the running request.
    +        running_ids = {
    +            id(a)
    +            for a in getattr(self, "_running_agents", {}).values()
    +            if a is not None and a is not _AGENT_PENDING_SENTINEL
    +        }
    +        if id(agent) in running_ids:
    +            return
    +
    +        try:
    +            threading.Thread(
    +                target=self._release_evicted_agent_soft,
    +                args=(agent,),
    +                daemon=True,
    +                name=f"agent-evict-{str(session_key)[:24]}",
    +            ).start()
    +        except Exception:
    +            # If we can't spawn a thread (interpreter shutdown), release
    +            # inline as a best-effort fallback.
    +            try:
    +                self._release_evicted_agent_soft(agent)
    +            except Exception:
    +                pass
     
         @staticmethod
         def _init_cached_agent_for_turn(agent: Any, interrupt_depth: int) -> None:
    ```
- **#41975: fix(gateway): close residual memory-leak sites under heavy scheduled workload**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `gateway/run.py`, `model_tools.py`, `tests/test_get_tool_definitions_cache_isolation.py`
- **#122925: fix(agent): close one-shot AIAgents on every exit path**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `batch_runner.py`, `hermes_cli/cli_commands_mixin.py`, `plugins/platforms/feishu/feishu_comment.py`, `tests/test_agent_close_lifecycle.py`, `tui_gateway/methods_prompt.py`
- **#10899: fix(gateway): close temporary agents after one-off tasks**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#11028`, `#10945`
- **#10945: fix(gateway): clean up temporary agent resources**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#11028`, `#10899`
- **#25318: fix(gateway): prevent memory leak from agent cache eviction and unbounded session data**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `3d029a53e`, `#41975`
- **#25332: fix: stop gateway memory leak on cached agent and session expiry**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`
- **#25377: fix(gateway): release cached agents on eviction**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `3d029a53ec327fd7a628cde900bf300ca9a0cc0f`
- **#50383: fix(tui-gateway): close AIAgent instances to release resources (#50197)**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`
- **#50403: fix(tui-gateway): close AIAgent instances to release resources (#50197)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `2fa2f42d41`, `94f3dbec9b`, `#122925`, `#50197`
- **#64284: fix: clean up evicted memory providers (salvage of #63703 + review fixes)**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`
  - files: `gateway/run.py`, `plugins/memory/__init__.py`, `plugins/memory/hindsight/__init__.py`, `run_agent.py`, `tests/agent/test_memory_provider.py`, `tests/gateway/test_agent_cache.py`, `tests/plugins/memory/test_hindsight_provider.py`, `tests/run_agent/test_shutdown_memory_provider_finalize_session.py`
- **#64644: fix(gateway): shut down memory provider on soft cache eviction (#64278)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#64284`, `#64278`, `#64644`, `#64284`, `#64284`

## Fix status

- Merged to main in PR #11028 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-09-25`.
- Subsystem: `agent`