# NousResearch/hermes-agent · httpx transport keepalive fix for streaming issues ## Open now - Nothing open on GitHub. These items address a problem where custom socket_options transport caused issues (including dropped chunked streaming with OpenResty/DeepSeek), proposing replacement with httpx pool-level keepalive expiry configuration. ## TL;DR - 3 member(s): 0 open issue(s), 1 closed issue(s), 0 open PR(s), 1 merged PR(s), 1 PR(s) closed without merging. - A fix was merged to main in PR #58766. Verify it addresses your case. - **Recommended:** read PR #58766 (and any competing PRs below) before filing. Confirm the merged change covers your symptom. ## Maintainer resolution verdict - Internal-tracker verdict over this cluster's issue members: `unresolved`. This is distinct from the fix status above. Weigh it yourself. ## How this cluster was grouped - Link evidence (`STRUCTURAL_SPANNING`): every member is connected by structural evidence: shared changed files, or a closing/superseded reference. - Pairwise link density `0.67`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked. ## Closed / duplicate issues - #54049: DeepSeek: OpenResty drops chunked streaming with custom httpx transport ## Prior & competing PRs - **#58766: fix(agent): replace socket_options transport with httpx pool-level keepalive expiry (#54550 salvage)**. merged to main · fix tier `verified` - maintainer direction: category `merged` - files: `agent/process_bootstrap.py`, `run_agent.py`, `tests/run_agent/test_create_openai_client_proxy_env.py` - fix hunks: - `agent/process_bootstrap.py` ```diff @@ -151,42 +151,51 @@ def build_keepalive_http_client( """Build an httpx client for OpenAI SDK calls with env-only proxy policy. Uses explicit ``HTTPS_PROXY`` / ``NO_PROXY`` env vars via - ``_get_proxy_for_base_url``. A custom transport disables httpx's default - ``trust_env`` path, so macOS system proxy settings from + ``_get_proxy_for_base_url``. Plain no-proxy mounts disable httpx's default + ``trust_env`` proxy path, so macOS system proxy settings from ``urllib.request.getproxies()`` (which omit the ExceptionsList) are not applied. Mirrors ``AIAgent._build_keepalive_http_client``. + Connection lifecycle is managed at the HTTP pool layer + (``keepalive_expiry=20.0`` reaps idle connections before reverse proxies' + typical 30-60 s timeouts) instead of the former custom + ``socket_options`` transport, which broke streaming behind reverse + proxies (#54049, #12952) and stalled TLS handshakes by stripping + ``TCP_NODELAY``. + ``verify`` is forwarded to httpx so auxiliary-client calls (compression, vision, web_extract, title generation, etc.) honor the same per-provider ``ssl_ca_cert`` / ``ssl_verify`` and ``HERMES_CA_BUNDLE`` settings the main - client uses. It is passed on the ``HTTPTransport`` (which owns the SSL - context when a custom transport is supplied) and, for the copilot branch - that has no custom transport, on the client itself. + client uses. It is passed on the client AND on the plain no-proxy mounts + (a mounted transport owns the SSL context for its scheme). """ try: import httpx - import socket - if "api.githubcopilot.com" in str(base_url or "").lower(): - client_cls = httpx.AsyncClient if async_mode else httpx.Client - return client_cls(verify=verify) + proxy = _get_proxy_for_base_url(base_url) - sock_opts = [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)] - if hasattr(socket, "TCP_KEEPIDLE"): - sock_opts.append((socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 30)) - sock_opts.append((socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)) - sock_opts.append((socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)) - elif hasattr(socket, "TCP_KEEPALIVE"): - sock_opts.append((socket.IPPROTO_TCP, socket.TCP_KEEPALIVE, 30)) + limits = httpx.Limits( + max_keepalive_connections=20, + max_connections=100, + keepalive_expiry=20.0, + ) + # Generous read=None for SSE streaming endpoints. + timeout = httpx.Timeout(connect=15.0, read=None, write=15.0, pool=10.0) - proxy = _get_proxy_for_base_url(base_url) transport_cls = httpx.AsyncHTTPTransport if async_mode else httpx.HTTPTransport client_cls = httpx.AsyncClient if async_mode else httpx.Client - # verify lives on the transport: httpx ignores the client-level - # ``verify`` when a custom ``transport=`` is supplied. + mounts = {} + if proxy is None: + mounts = { + "http://": transport_cls(verify=verify), + "https://": transport_cls(verify=verify), + } return client_cls( - transport=transport_cls(socket_options=sock_opts, verify=verify), + limits=limits, + timeout=timeout, proxy=proxy, + mounts=mounts or None, + verify=verify, ) except Exception: return None ``` - `run_agent.py` ```diff @@ -3896,30 +3896,70 @@ def _is_openai_client_closed(client: Any) -> bool: @staticmethod def _build_keepalive_http_client(base_url: str = "", *, verify: Any = True) -> Any: + """Build an httpx.Client with proactive idle-connection reaping. + + Previously this method injected a custom ``httpx.HTTPTransport`` + with ``socket_options`` (``SO_KEEPALIVE``, ``TCP_KEEPIDLE``, …) to + prevent CLOSE-WAIT accumulation on long-lived connections (#10324). + + That approach broke streaming for providers behind reverse proxies + (OpenResty, Cloudflare, etc.) because the custom socket options + conflict with the proxy's chunked-transfer handling (#54049, + #12952). It also stripped ``TCP_NODELAY``, stalling TLS handshakes + and SSE encoding. + + The fix moves connection lifecycle management from the socket layer + to the HTTP pool layer: ``keepalive_expiry=20.0`` tells httpx to + close idle pooled connections *before* a reverse proxy's typical + 30–60 s timeout drops them, preventing CLOSE-WAIT accumulation + without modifying socket options. The default httpx transport + preserves OS TCP defaults (including ``TCP_NODELAY``). + + ``verify`` carries per-provider ``ssl_ca_cert`` / ``ssl_verify`` and + ``HERMES_CA_BUNDLE`` settings. It is passed on the client AND on + the plain no-proxy mounts (a mounted transport owns the SSL context + for its scheme). + """ try: import httpx as _httpx - import socket as _socket - - if "api.githubcopilot.com" in str(base_url or "").lower(): - return _httpx.Client(verify=verify) - - _sock_opts = [(_socket.SOL_SOCKET, _socket.SO_KEEPALIVE, 1)] - if hasattr(_socket, "TCP_KEEPIDLE"): - _sock_opts.append((_socket.IPPROTO_TCP, _socket.TCP_KEEPIDLE, 30)) - _sock_opts.append((_socket.IPPROTO_TCP, _socket.TCP_KEEPINTVL, 10)) - _sock_opts.append((_socket.IPPROTO_TCP, _socket.TCP_KEEPCNT, 3)) - elif hasattr(_socket, "TCP_KEEPALIVE"): - _sock_opts.append((_socket.IPPROTO_TCP, _socket.TCP_KEEPALIVE, 30)) - # When a custom transport is provided, httpx won't auto-read proxy - # from env vars (allow_env_proxies = trust_env and transport is None). - # Explicitly read proxy settings while still honoring NO_PROXY for - # loopback / local endpoints such as a locally hosted sub2api. + + # Explicitly read proxy settings so requests route through + # HTTP_PROXY / HTTPS_PROXY / NO_PROXY correctly. _proxy = _get_proxy_for_base_url(base_url) - # verify lives on the transport: httpx ignores the client-level - # ``verify`` when a custom ``transport=`` is supplied. + + # Proactive pool reaping: close idle connections at 20 s, + # before reverse proxies (30–60 s typical) send FIN and + # cause CLOSE-WAIT accumulation. + _limits = _httpx.Limits( + max_keepalive_connections=20, + max_connections=100, + keepalive_expiry=20.0, + ) + + # Timeouts: generous read=None for SSE streaming endpoints. + _timeout = _httpx.Timeout( + connect=15.0, + read=None, + write=15.0, + pool=10.0, + ) + + # When _proxy is None (NO_PROXY bypass or no proxy configured), + # mount plain transports to prevent httpx from reading env proxy + # vars and creating an HTTPProxy mount that would bypass our + # NO_PROXY resolution. + _mounts = {} + if _proxy is None: + _mounts = { + "http://": _httpx.HTTPTransport(verify=verify), + "https://": _httpx.HTTPTransport(verify=verify), + } return _httpx.Client( - transport=_httpx.HTTPTransport(socket_options=_sock_opts, verify=verify), + limits=_limits, + timeout=_timeout, proxy=_proxy, + mounts=_mounts or None, + verify=verify, ) except Exception: return None ``` - `tests/run_agent/test_create_openai_client_proxy_env.py` ```diff @@ -1,22 +1,14 @@ """Regression guard: _create_openai_client must honor HTTP(S)_PROXY env vars. -When #11277 re-landed TCP keepalives, ``_create_openai_client`` began passing -a custom ``transport=httpx.HTTPTransport(...)`` to ``httpx.Client``. httpx only -auto-reads ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``ALL_PROXY`` when -``transport is None`` (see ``Client.__init__``: -``allow_env_proxies = trust_env and transport is None``). As a result, proxy -env vars were silently ignored for the primary chat client, causing requests -to bypass local proxies (Clash, corporate egress, etc.) and hit upstream -directly from the raw interface. - -For users on WSL2 + Clash TUN this surfaced as Cloudflare ``cf-mitigated: -challenge`` 403s against ``chatgpt.com/backend-api/codex`` once they upgraded -past #11277. The fix forwards the proxy URL explicitly to ``httpx.Client`` -while keeping the keepalive-enabled transport in place. - -This test pins that the constructed ``httpx.Client`` mounts an ``HTTPProxy`` -pool when a proxy env var is set, AND that the socket-level keepalive -transport is still installed on the no-proxy default path. +The keepalive client now uses ``httpx.Limits(keepalive_expiry=20.0)`` +instead of a custom ``httpx.HTTPTransport(socket_options=...)`` to +prevent CLOSE-WAIT accumulation. This avoids breaking streaming for +providers behind reverse proxies (#54049, #12952) while still reaping +idle connections before a proxy's timeout drops them. + +This test pins that the constructed ``httpx.Client`` mounts an +``HTTPProxy`` pool when a proxy env var is set, and that no +custom socket-options transport is used (default httpx transport). """ from unittest.mock import patch ``` ```diff @@ -117,8 +109,8 @@ def test_create_openai_client_routes_via_proxy_when_env_set(mock_openai, monkeyp @patch("run_agent.OpenAI") def test_create_openai_client_no_proxy_when_env_unset(mock_openai, monkeypatch): - """Without proxy env vars, the keepalive transport must still be installed - and no HTTPProxy mount should exist.""" + """Without proxy env vars, no HTTPProxy mount should exist and + no custom socket-options transport should be installed.""" for key in ("HTTPS_PROXY", "HTTP_PROXY", "ALL_PROXY", "https_proxy", "http_proxy", "all_proxy"): monkeypatch.delenv(key, raising=False) ``` ```diff @@ -147,7 +139,8 @@ def test_create_openai_client_no_proxy_when_env_unset(mock_openai, monkeypatch): @patch("run_agent.OpenAI") def test_create_openai_client_uses_plain_httpx_client_for_copilot(mock_openai, monkeypatch): - """Copilot Claude chat-completions rejects the custom socket-options transport.""" + """All providers now use a standard httpx.Client (no custom socket-options + transport) so Copilot Claude chat-completions works without a host bypass.""" for key in ("HTTPS_PROXY", "HTTP_PROXY", "ALL_PROXY", "https_proxy", "http_proxy", "all_proxy"): monkeypatch.delenv(key, raising=False) ``` - **#54550: fix(agent): replace custom socket_options transport with httpx pool-level keepalive expiry**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `51c1ba697`, `#58766` ## Fix status - Merged to main in PR #58766 (cited above). No release data is tracked. Verify downstream. - Fix last verified (merged) `2026-07-05`. - Subsystem: `agent`