# NousResearch/hermes-agent · Retry backoff, rate-limit handling, and local endpoint timeouts

## Open now

- 6 issues, 5 PRs open on GitHub.
- #5570: feat: configurable max API retries + stream retries with smarter backoff
- #20346: \[Bug\]: using DNS for local provider is not addressed in recent fix
- #44468: \[Bug\]: \`hermes send\` to Discord drops remaining chunks on a 429 — no retry in the standalone send path (Telegram has one)
- #49031: \[Feature Request\] Configurable retry backoff for 429 rate-limit errors
- #54875: Configurable retry backoff parameters (base_delay, max_delay) for HTTP 429
- #55540: \[Feature\]: Longer/configurable backoff for HTTP 503/529 provider overload (parity with Z.AI path)
- PR #8486: feat: configurable max API retries + stream retries with smarter backoff
- PR #13519: Amindadgar PR#8486 retry ci fix
- PR #20359: fix(agent): treat mDNS/.lan/unqualified hostnames as local endpoints (#20346)
- PR #102551: feat(agent): configurable rate-limit retry policy for 429s
- PR #109329: fix(discord): retry standalone sends after rate limits

These items address HTTP 429/503/529 retry logic with configurable backoff and user-friendly messaging, Discord chunk-drop bugs on rate limits, and stale-stream timeout behavior for local/Docker/mDNS provider endpoints.

## TL;DR

- 33 member(s): 6 open issue(s), 6 closed issue(s), 5 open PR(s), 6 merged PR(s), 10 PR(s) closed without merging.
- A fix was merged to main in PR #3809. Verify it addresses your case.
- **Recommended:** read PR #3809 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-6368,pr-9248`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text.
- Members outside that structural core may belong to a different underlying problem.
- Pairwise link density `0.05`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Open related issues

- #5570: feat: configurable max API retries + stream retries with smarter backoff
- #20346: \[Bug\]: using DNS for local provider is not addressed in recent fix
- #44468: \[Bug\]: \`hermes send\` to Discord drops remaining chunks on a 429 — no retry in the standalone send path (Telegram has one)
- #49031: \[Feature Request\] Configurable retry backoff for 429 rate-limit errors
- #54875: Configurable retry backoff parameters (base_delay, max_delay) for HTTP 429
- #55540: \[Feature\]: Longer/configurable backoff for HTTP 503/529 provider overload (parity with Z.AI path)

## Closed / duplicate issues

- #1826: \[UX\] Rate limit errors (429) should show user-friendly message with configurable retry instead of traceback
- #4657: \[Feature\]: Configurable server retry interval
- #5889: \[Bug\]:Hermes reconnects after 180s of provider silence even though oMLX is still processing
- #7069: \[Bug\]: Infinite retry loop caused by stream stale timeout during local LLM prefill phase
- #7905: is_local_endpoint misses Docker/Podman DNS names — stale timeout fires on local LLM proxies
- #29702: Feature Request: Configurable model provider retry interval / backoff

## Prior & competing PRs

- **#3809: fix(agent): user-friendly 429 rate limit messages with Retry-After support**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `run_agent.py`
  - fix hunks:
  - `run_agent.py`

    ```diff
    @@ -7229,7 +7229,10 @@ def _stop_spinner():
                                 retry_count = 0
                                 continue
                             _final_summary = self._summarize_api_error(api_error)
    -                        self._vprint(f"{self.log_prefix}❌ Max retries ({max_retries}) exceeded. Giving up.", force=True)
    +                        if is_rate_limited:
    +                            self._vprint(f"{self.log_prefix}❌ Rate limit persisted after {max_retries} retries. Please try again later.", force=True)
    +                        else:
    +                            self._vprint(f"{self.log_prefix}❌ Max retries ({max_retries}) exceeded. Giving up.", force=True)
                             self._vprint(f"{self.log_prefix}   💀 Final error: {_final_summary}", force=True)
     
                             # Detect SSE stream-drop pattern (e.g. "Network
    ```

    ```diff
    @@ -7289,8 +7292,22 @@ def _stop_spinner():
                                 "error": _final_summary,
                             }
     
    -                    wait_time = min(2 ** retry_count, 60)  # Exponential backoff: 2s, 4s, 8s, 16s, 32s, 60s, 60s
    -                    self._emit_status(f"⏳ Retrying in {wait_time}s (attempt {retry_count}/{max_retries})...")
    +                    # For rate limits, respect the Retry-After header if present
    +                    _retry_after = None
    +                    if is_rate_limited:
    +                        _resp_headers = getattr(getattr(api_error, "response", None), "headers", None)
    +                        if _resp_headers and hasattr(_resp_headers, "get"):
    +                            _ra_raw = _resp_headers.get("retry-after") or _resp_headers.get("Retry-After")
    +                            if _ra_raw:
    +                                try:
    +                                    _retry_after = min(int(_ra_raw), 120)  # Cap at 2 minutes
    +                                except (TypeError, ValueError):
    +                                    pass
    +                    wait_time = _retry_after if _retry_after else min(2 ** retry_count, 60)
    +                    if is_rate_limited:
    +                        self._emit_status(f"⏱️ Rate limit reached. Waiting {wait_time}s before retry (attempt {retry_count + 1}/{max_retries})...")
    +                    else:
    +                        self._emit_status(f"⏳ Retrying in {wait_time}s (attempt {retry_count}/{max_retries})...")
                         logger.warning(
                             "Retrying API call in %ss (attempt %s/%s) %s error=%s",
                             wait_time,
    ```
- **#6368: fix(agent): disable stale stream timeout for local providers**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `run_agent.py`
- **#7950: fix: is_local_endpoint misses Docker/Podman DNS names**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `agent/model_metadata.py`, `tests/agent/test_local_stream_timeout.py`, `tests/tools/test_tool_result_storage.py`, `tools/tool_result_storage.py`
  - fix hunks:
  - `agent/model_metadata.py`

    ```diff
    @@ -179,6 +179,12 @@ def _strip_provider_prefix(model: str) -> str:
     
     # Local server hostnames / address patterns
     _LOCAL_HOSTS = ("localhost", "127.0.0.1", "::1", "0.0.0.0")
    +# Docker / Podman / Lima DNS names that resolve to the host machine
    +_CONTAINER_LOCAL_SUFFIXES = (
    +    ".docker.internal",
    +    ".containers.internal",
    +    ".lima.internal",
    +)
     
     
     def _normalize_base_url(base_url: str) -> str:
    ```

    ```diff
    @@ -254,6 +260,9 @@ def is_local_endpoint(base_url: str) -> bool:
             return False
         if host in _LOCAL_HOSTS:
             return True
    +    # Docker / Podman / Lima internal DNS names (e.g. host.docker.internal)
    +    if any(host.endswith(suffix) for suffix in _CONTAINER_LOCAL_SUFFIXES):
    +        return True
         # RFC-1918 private ranges and link-local
         import ipaddress
         try:
    ```
  - `tests/agent/test_local_stream_timeout.py`

    ```diff
    @@ -22,6 +22,9 @@ class TestLocalStreamReadTimeout:
             "http://0.0.0.0:5000",
             "http://192.168.1.100:8000",
             "http://10.0.0.5:1234",
    +        "http://host.docker.internal:11434",
    +        "http://host.containers.internal:11434",
    +        "http://host.lima.internal:11434",
         ])
         def test_local_endpoint_bumps_read_timeout(self, base_url):
             """Local endpoint + default timeout -> bumps to base_timeout."""
    ```

    ```diff
    @@ -68,3 +71,38 @@ def test_empty_base_url_keeps_default(self):
                 if _stream_read_timeout == 120.0 and base_url and is_local_endpoint(base_url):
                     _stream_read_timeout = _base_timeout
                 assert _stream_read_timeout == 120.0
    +
    +
    +class TestIsLocalEndpoint:
    +    """Direct unit tests for is_local_endpoint."""
    +
    +    @pytest.mark.parametrize("url", [
    +        "http://localhost:11434",
    +        "http://127.0.0.1:8080",
    +        "http://0.0.0.0:5000",
    +        "http://[::1]:11434",
    +        "http://192.168.1.100:8000",
    +        "http://10.0.0.5:1234",
    +        "http://172.17.0.1:11434",
    +    ])
    +    def test_classic_local_addresses(self, url):
    +        assert is_local_endpoint(url) is True
    +
    +    @pytest.mark.parametrize("url", [
    +        "http://host.docker.internal:11434",
    +        "http://host.docker.internal:8080/v1",
    +        "http://gateway.docker.internal:11434",
    +        "http://host.containers.internal:11434",
    +        "http://host.lima.internal:11434",
    +    ])
    +    def test_container_dns_names(self, url):
    +        assert is_local_endpoint(url) is True
    +
    +    @pytest.mark.parametrize("url", [
    +        "https://api.openai.com",
    +        "https://openrouter.ai/api",
    +        "https://api.anthropic.com",
    +        "https://evil.docker.internal.example.com",
    +    ])
    +    def test_remote_endpoints(self, url):
    +        assert is_local_endpoint(url) is False
    ```
  - `tests/tools/test_tool_result_storage.py`

    ```diff
    @@ -124,6 +124,34 @@ def test_uses_parent_dir_of_remote_path(self):
             cmd = env.execute.call_args[0][0]
             assert "mkdir -p /data/data/com.termux/files/usr/tmp/hermes-results" in cmd
     
    +    def test_path_with_spaces_is_quoted(self):
    +        env = MagicMock()
    +        env.execute.return_value = {"output": "", "returncode": 0}
    +        remote_path = "/tmp/hermes results/abc file.txt"
    +        _write_to_sandbox("content", remote_path, env)
    +        cmd = env.execute.call_args[0][0]
    +        assert "'/tmp/hermes results'" in cmd
    +        assert "'/tmp/hermes results/abc file.txt'" in cmd
    +
    +    def test_shell_metacharacters_neutralized(self):
    +        """Paths with shell metacharacters must be quoted to prevent injection."""
    +        env = MagicMock()
    +        env.execute.return_value = {"output": "", "returncode": 0}
    +        malicious_path = "/tmp/hermes-results/$(whoami).txt"
    +        _write_to_sandbox("content", malicious_path, env)
    +        cmd = env.execute.call_args[0][0]
    +        # The $() must not appear unquoted — shlex.quote wraps it
    +        assert "'/tmp/hermes-results/$(whoami).txt'" in cmd
    +
    +    def test_semicolon_injection_neutralized(self):
    +        env = MagicMock()
    +        env.execute.return_value = {"output": "", "returncode": 0}
    +        malicious_path = "/tmp/x; rm -rf /; echo .txt"
    +        _write_to_sandbox("content", malicious_path, env)
    +        cmd = env.execute.call_args[0][0]
    +        # The semicolons must be inside quotes, not acting as command separators
    +        assert "'/tmp/x; rm -rf /; echo .txt'" in cmd
    +
     
     class TestResolveStorageDir:
         def test_defaults_to_storage_dir_without_env(self):
    ```
  - `tools/tool_result_storage.py`

    ```diff
    @@ -24,6 +24,7 @@
     
     import logging
     import os
    +import shlex
     import uuid
     
     from tools.budget_config import (
    ```

    ```diff
    @@ -79,7 +80,7 @@ def _write_to_sandbox(content: str, remote_path: str, env) -> bool:
         marker = _heredoc_marker(content)
         storage_dir = os.path.dirname(remote_path)
         cmd = (
    -        f"mkdir -p {storage_dir} && cat > {remote_path} << '{marker}'\n"
    +        f"mkdir -p {shlex.quote(storage_dir)} && cat > {shlex.quote(remote_path)} << '{marker}'\n"
             f"{content}\n"
             f"{marker}"
         )
    ```
- **#8021: fix(agent): detect Docker/Podman DNS names in is_local_endpoint**. closed without merge · fix tier `suggested`
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#7906`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#9248: fix(local): recognize unqualified hostnames as local endpoints**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `agent/model_metadata.py`, `tests/agent/test_local_stream_timeout.py`
- **#44558: fix(tools): retry Discord send on 429 rate-limit (#44468)**. closed without merge · fix tier `suggested`
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#44488`, `#44468`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#77607: fix(agent): exponential backoff for rate-limit fallback cooldown (#30223 salvage)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `agent/agent_runtime_helpers.py`, `agent/chat_completion_helpers.py`, `tests/run_agent/test_24996_fallback_exhaustion_cooldown.py`
- **#116353: Transient provider outages no longer end the turn: bounded auto-recovery ladder after retries and fallback, visible on CLI/TUI/gateway/API/cron (#85426, #107307)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `agent/AGENTS.md`, `agent/agent_init.py`, `agent/turn_api_error.py`, `agent/turn_recovery_autorecover.py`, `agent/turn_retry_state.py`, `cli-config.yaml.example`, `gateway/platforms/api_server.py`, `gateway/platforms/api_server_openai_routes.py`, `hermes_cli/config_defaults.py`, `tests/agent/test_turn_recovery_autorecover.py`, `tests/agent/test_turn_retry_state.py`, `tests/gateway/test_api_server_status_stream.py`, `website/docs/user-guide/configuration.md`
- **#1835: fix(agent): user-friendly 429 rate limit messages with countdown and Retry-After support**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `#3809`
- **#6123: fix(run_agent): disable stale timeout for local providers (#5889)**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#6368`, `#5889`
- **#7431: fix(agent): add exponential backoff to inner streaming retry loop**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `985e19c110f19e1131b52f78c6505e2a06837127`
- **#7906: fix: is_local_endpoint misses Docker/Podman DNS names**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#7950`, `#9248`
- **#8486: feat: configurable max API retries + stream retries with smarter backoff**. open
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `165b2e481afa1cd7385c5c9f9ebf33eb4a524131`, `#14730`
  - files: `cli.py`, `run_agent.py`, `tests/test_api_retry_config.py`
- **#13519: Amindadgar PR#8486 retry ci fix**. open
  - maintainer direction: category `merged_via`; implication `superseded`
  - files: `.gitignore`, `cli-config.yaml.example`, `cli.py`, `docs/acp-setup.md`, `gateway/run.py`, `hermes_cli/config.py`, `run_agent.py`, `tests/cli/test_cli_init.py`, `tests/hermes_cli/test_config.py`, `tests/run_agent/test_run_agent.py`, `tests/test_api_retry_config.py`
- **#20359: fix(agent): treat mDNS/.lan/unqualified hostnames as local endpoints (#20346)**. open
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `#9248`
  - files: `agent/model_metadata.py`, `scripts/release.py`, `tests/agent/test_local_stream_timeout.py`
- **#30223: fix: exponential backoff for rate-limit fallback cooldown**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#77607`
- **#44488: fix(discord): retry 429s in the standalone send path, honoring retry_after**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#79564`
- **#44542: fix(discord): retry standalone send on 429 rate-limit instead of dropping chunks**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#44488`
- **#56037: feat(agent): make overload (503/529) backoff configurable**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `059b3d416f`, `#116353`
- **#102551: feat(agent): configurable rate-limit retry policy for 429s**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#46912`, `#49031`, `#70738`
  - files: `agent/agent_init.py`, `agent/agent_runtime_helpers.py`, `agent/conversation_loop.py`, `agent/rate_limit_policy.py`, `cli-config.yaml.example`, `contributors/emails/[redacted:email]`, `hermes_cli/config_defaults.py`, `tests/run_agent/test_rate_limit_pool_hold.py`, `tests/run_agent/test_rate_limit_retry_policy.py`, `website/docs/user-guide/configuration.md`
- **#109329: fix(discord): retry standalone sends after rate limits**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#44468`, `#44488`, `#44542`, `#79564`, `#109384`, `#35647`
  - files: `plugins/platforms/discord/adapter.py`, `tests/gateway/test_discord_send.py`

## Fix status

- Merged to main in PR #3809 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-09-19`.
- Subsystem: `agent`