{"repo":"NousResearch/hermes-agent","public_id":"1034","display_label":"Configurable per-task fallback chains for auxiliary models","summary":"These issues and PRs focus on adding configurable fallback provider chains to auxiliary model slots (vision, compression, STT, summarization, web extraction, etc.) so that failures or auth errors transparently route to backup models. Several members also address surfacing clearer error messages and graceful recovery when primary paths fail.","subsystem":"agent","counts":{"members":14,"open_issues":1,"open_issues_distinct":1,"open_prs":2,"closed_issues":6,"merged_prs":2,"attempt_prs":5},"landed_fix":{"status":"merged_to_main","basis":"structural","pr":1338},"resolution":{"state":"partially_resolved","by":"pr-27625"},"freshness":{"state":"verified","last_verified_at":"2026-05-18T00:15:31+00:00","is_stale":false},"grouping":{"backbone":"STRUCTURAL_PARTIAL","link_density":0.13142857142857145},"fix_edges":[{"issue_number":1034,"pr_number":1338,"tier":"verified","basis":"superseded_ref","confidence":0.9,"recoverable_from_timeline":false},{"issue_number":26803,"pr_number":27625,"tier":"verified","basis":"closing_ref","confidence":0.95,"recoverable_from_timeline":true},{"issue_number":26882,"pr_number":27625,"tier":"verified","basis":"closing_ref","confidence":0.95,"recoverable_from_timeline":true}],"members":[{"kind":"issue","number":1034,"title":"bug(vision): image analysis should recover gracefully when primary vision path fails","url":null,"state":"closed"},{"kind":"issue","number":17067,"title":"feat: fallback models for auxiliary/web_extract","url":null,"state":"closed"},{"kind":"issue","number":22201,"title":"Per-auxiliary fallback_providers for independent task resilience","url":null,"state":"open"},{"kind":"issue","number":26882,"title":"[Feature] Configurable fallback chains for auxiliary tasks","url":null,"state":"closed"},{"kind":"issue","number":27298,"title":"feat: General per-task fallback chains for ALL auxiliary models (vision, compression, STT, summarization, etc.)","url":null,"state":"closed"},{"kind":"issue","number":32408,"title":"Proposal: consolidate ~12 auxiliary fallback PRs into per-task fallback_providers","url":null,"state":"closed"},{"kind":"issue","number":51814,"title":"Feature Request: Fallback providers for auxiliary model slots","url":null,"state":"closed"},{"kind":"pr","number":1038,"title":"fix(vision): surface actual error reason instead of generic message","url":null,"state":"closed","close_reason":{"refs":[{"kind":"pr","value":"#1338","number":1338}],"category":"merged_via","implication":"superseded"},"merged":false},{"kind":"pr","number":1338,"title":"fix(vision): surface actual error reason instead of generic message","url":null,"state":"merged","close_reason":{"category":"merged","implication":"superseded"},"merged":true,"is_fix_pr":true,"fix_tier":"verified","files_changed":["tools/vision_tools.py"]},{"kind":"pr","number":21190,"title":"fix(auxiliary): fallback on auth errors in auto mode","url":null,"state":"closed","close_reason":{"refs":[{"kind":"commit","value":"46e18804a","number":0},{"kind":"pr","value":"#53900","number":53900}],"category":"merged_via","implication":"superseded"},"merged":false},{"kind":"pr","number":26998,"title":"feat(auxiliary): add configurable fallback chains for auxiliary tasks (#26882)","url":null,"state":"closed","close_reason":{"refs":[{"kind":"commit","value":"a57424683","number":0},{"kind":"pr","value":"#27625","number":27625},{"kind":"pr","value":"#26811","number":26811}],"category":"superseded_by_pr","implication":"superseded"},"merged":false},{"kind":"pr","number":27625,"title":"feat(auxiliary): layered fallback (chain → main agent) + capacity-error gate fix","url":null,"state":"merged","close_reason":{"category":"merged","implication":"superseded"},"merged":true,"is_fix_pr":true,"fix_tier":"verified","files_changed":["agent/auxiliary_client.py","scripts/release.py","tests/agent/test_auxiliary_client.py","website/docs/user-guide/features/fallback-providers.md"],"fix_hunks":[{"file":"agent/auxiliary_client.py","hunks":["@@ -2087,7 +2087,13 @@ def _is_payment_error(exc: Exception) -> bool:\n     \"\"\"Detect payment/credit/quota exhaustion errors.\n \n     Returns True for HTTP 402 (Payment Required) and for 429/other errors\n-    whose message indicates billing exhaustion rather than rate limiting.\n+    whose message indicates billing exhaustion or daily quota exhaustion\n+    rather than transient rate limiting.\n+\n+    Daily token quota errors (e.g. Bedrock \"Too many tokens per day\",\n+    Vertex AI \"quota exceeded\") are functionally equivalent to credit\n+    exhaustion — the provider cannot serve the request until the quota\n+    resets — and should trigger the same provider-fallback logic.\n     \"\"\"\n     status = getattr(exc, \"status_code\", None)\n     if status == 402:","@@ -2095,10 +2101,19 @@ def _is_payment_error(exc: Exception) -> bool:\n     err_lower = str(exc).lower()\n     # OpenRouter and other providers include \"credits\" or \"afford\" in 402 bodies,\n     # but sometimes wrap them in 429 or other codes.\n+    # Daily quota exhaustion from Bedrock, Vertex AI, and similar providers\n+    # uses different language but is semantically identical to credit exhaustion.\n     if status in {402, 429, None}:\n-        if any(kw in err_lower for kw in (\"credits\", \"insufficient funds\",\n-                                           \"can only afford\", \"billing\",\n-                                           \"payment required\")):\n+        if any(kw in err_lower for kw in (\n+            \"credits\", \"insufficient funds\",\n+            \"can only afford\", \"billing\",\n+            \"payment required\",\n+            # Daily / monthly quota exhaustion keywords\n+            \"quota exceeded\", \"quota_exceeded\",\n+            \"too many tokens per day\", \"daily limit\",\n+            \"tokens per day\", \"daily quota\",\n+            \"resource exhausted\",  # Vertex AI / gRPC quota errors\n+        )):\n             return True\n     return False\n ","@@ -2579,6 +2594,133 @@ def _try_payment_fallback(\n     return None, None, \"\"\n \n \n+def _try_main_agent_model_fallback(\n+    failed_provider: str,\n+    task: str = None,\n+    reason: str = \"error\",\n+) -> Tuple[Optional[Any], Optional[str], str]:\n+    \"\"\"Last-resort fallback to the user's main agent provider + model.\n+\n+    Used after the configured fallback_chain is exhausted (or empty) for\n+    users with an explicit auxiliary provider.  This is the \"safety net\"\n+    layer: if nothing the user asked for can serve the request, try the\n+    main chat model before giving up.\n+\n+    Skips when the failed provider already IS the main provider (no point\n+    retrying the same backend that just failed).\n+\n+    Returns:\n+        (client, model, provider_label) or (None, None, \"\") if no fallback.\n+    \"\"\"\n+    main_provider = (_read_main_provider() or \"\").strip()\n+    main_model = (_read_main_model() or \"\").strip()\n+    if not main_provider or not main_model or main_provider.lower() in {\"auto\", \"\"}:\n+        return None, None, \"\"\n+\n+    skip = (failed_provider or \"\").lower().strip()\n+    if main_provider.lower() == skip:\n+        # The thing that failed IS the main model — nothing to fall back to.\n+        return None, None, \"\"\n+    if _is_provider_unhealthy(main_provider):\n+        _log_skip_unhealthy(main_provider, task)\n+        return None, None, \"\"\n+\n+    try:\n+        client, resolved_model = resolve_provider_client(\n+            provider=main_provider, model=main_model,\n+        )\n+    except Exception:\n+        client, resolved_model = None, None\n+\n+    if client is None:\n+        return None, None, \"\"\n+\n+    label = f\"main-agent({main_provider})\"\n+    logger.info(\n+        \"Auxiliary %s: %s on %s — falling back to main agent model %s (%s)\",\n+        task or \"call\", reason, failed_provider, label, resolved_model or main_model,\n+    )\n+    return client, resolved_model or main_model, label\n+\n+\n+def _try_configured_fallback_chain(\n+    task: str,\n+    failed_provider: str,\n+    reason: str = \"error\",\n+) -> Tuple[Optional[Any], Optional[str], str]:\n+    \"\"\"Try user-configured fallback_chain for a specific auxiliary task.\n+\n+    Reads auxiliary.<task>.fallback_chain from config.yaml and tries each\n+    entry in order.  Each entry must have at least ``provider``; ``model``,\n+    ``base_url``, and ``api_key`` are optional.\n+\n+    Returns:\n+        (client, model, provider_label) or (None, None, \"\") if no fallback.\n+    \"\"\"\n+    if not task:\n+        return None, None, \"\"\n+\n+    task_config = _get_auxiliary_task_config(task)\n+    chain = task_config.get(\"fallback_chain\")\n+    if not chain or not isinstance(chain, list):\n+        return None, None, \"\"\n+\n+    skip = failed_provider.lower().strip()\n+    tried = []\n+\n+    for i, entry in enumerate(chain):\n+        if not isinstance(entry, dict):\n+            continue\n+        fb_provider = str(entry.get(\"provider\", \"\")).strip()\n+        if not fb_provider or fb_provider.lower() == skip:\n+            continue\n+        fb_model = str(entry.get(\"model\", \"\")).strip() or None\n+        fb_base_url = str(entry.get(\"base_url\", \"\")).strip() or None\n+        fb_api_key = [redacted:secret-assignment]\"api_key\", \"\")).strip() or None\n+\n+        label = f\"fallback_chain[{i}]({fb_provider})\"\n+\n+        try:\n+            fb_client = _resolve_single_provider(\n+                fb_provider, fb_model, fb_base_url, fb_api_key)\n+        except Exception:\n+            fb_client = None\n+\n+        if fb_client is not None:\n+            logger.info(\n+                \"Auxiliary %s: %s on %s — configured fallback to %s (%s)\",\n+                task, reason, failed_provider, label, fb_model or \"default\",\n+            )\n+            return fb_client, fb_model, label\n+        tried.append(label)\n+\n+    if tried:\n+        logger.debug(\n+            \"Auxiliary %s: configured fallback_chain exhausted (tried: %s)\",\n+            task, \", \".join(tried),\n+        )\n+    return None, None, \"\"\n+\n+\n+def _resolve_single_provider(\n+    provider: str,\n+    model: Optional[str] = None,\n+    base_url: Optional[str] = None,\n+    api_key: [redacted:secret-assignment] = None,\n+) -> Optional[Any]:\n+    \"\"\"Resolve a single provider entry from fallback_chain to an OpenAI client.\n+\n+    Uses the existing provider resolution infrastructure where possible.\n+    \"\"\"\n+    # Reuse resolve_provider_client which handles provider→client mapping\n+    client, resolved_model = resolve_provider_client(\n+        provider=provider,\n+        model=model,\n+        base_url=base_url,\n+        api_key=[redacted:secret-assignment]\n+    )\n+    return client\n+\n def _resolve_auto(main_runtime: Optional[Dict[str, Any]] = None) -> Tuple[Optional[OpenAI], Optional[str]]:\n     \"\"\"Full auto-detection chain.\n ","@@ -4526,11 +4668,17 @@ def call_llm(\n             or _is_connection_error(first_err)\n             or _is_rate_limit_error(first_err)\n         )\n-        # Only try alternative providers when the user didn't explicitly\n-        # configure this task's provider.  Explicit provider = hard constraint;\n-        # auto (the default) = best-effort fallback chain.  (#7559)\n+        # Respect explicit provider choice for transient errors (auth, request\n+        # validation, etc.) but allow fallback when the provider clearly cannot\n+        # serve the request due to capacity: payment/quota exhaustion and\n+        # connection failures are capacity problems, not request constraints.\n+        # See #26803: daily token quota (429 + \"too many tokens per day\") must\n+        # fall back just like a 402 credit error.\n         is_auto = resolved_provider in {\"auto\", \"\", None}\n-        if should_fallback and is_auto:\n+        # Capacity errors bypass the explicit-provider gate: the provider\n+        # literally cannot serve this request regardless of user intent.\n+        is_capacity_error = _is_payment_error(first_err) or _is_connection_error(first_err)\n+        if should_fallback and (is_auto or is_capacity_error):\n             if _is_payment_error(first_err):\n                 reason = \"payment error\"\n                 # Resolve the actual provider label (resolved_provider may be","@@ -4546,8 +4694,24 @@ def call_llm(\n                 reason = \"connection error\"\n             logger.info(\"Auxiliary %s: %s on %s (%s), trying fallback\",\n                         task or \"call\", reason, resolved_provider, first_err)\n-            fb_client, fb_model, fb_label = _try_payment_fallback(\n-                resolved_provider, task, reason=reason)\n+\n+            # Fallback order (#26882, #26803):\n+            #   1. User-configured fallback_chain (per-task) if set\n+            #   2. Main agent model (last-resort safety net)\n+            # For auto users (no explicit aux provider), use the full\n+            # auto-detection chain instead — its Step 1 IS the main agent\n+            # model, so users on `auto` already get main-model fallback.\n+            fb_client, fb_model, fb_label = (None, None, \"\")\n+            if is_auto:\n+                fb_client, fb_model, fb_label = _try_payment_fallback(\n+                    resolved_provider, task, reason=reason)\n+            else:\n+                fb_client, fb_model, fb_label = _try_configured_fallback_chain(\n+                    task, resolved_provider or \"auto\", reason=reason)\n+                if fb_client is None:\n+                    fb_client, fb_model, fb_label = _try_main_agent_model_fallback(\n+                        resolved_provider, task, reason=reason)\n+\n             if fb_client is not None:\n                 fb_kwargs = _build_call_kwargs(\n                     fb_label, fb_model, messages,","@@ -4557,6 +4721,14 @@ def call_llm(\n                     base_url=str(getattr(fb_client, \"base_url\", \"\") or \"\"))\n                 return _validate_llm_response(\n                     fb_client.chat.completions.create(**fb_kwargs), task)\n+            # All fallback layers exhausted — emit a single user-visible\n+            # warning so the operator knows aux task is about to fail.\n+            # (#26882) The error itself is re-raised below.\n+            logger.warning(\n+                \"Auxiliary %s: %s on %s and all fallbacks exhausted \"\n+                \"(fallback_chain + main agent model). Raising original error.\",\n+                task or \"call\", reason, resolved_provider,\n+            )\n         # Connection/timeout errors leave the cached client poisoned (closed\n         # httpx transport, half-read stream, dead async loop).  Drop it from\n         # the cache regardless of whether we found a fallback above so the","@@ -4858,8 +5030,12 @@ async def async_call_llm(\n             or _is_connection_error(first_err)\n             or _is_rate_limit_error(first_err)\n         )\n+        # Capacity errors (payment/quota/connection) bypass the explicit-provider\n+        # gate — the provider cannot serve the request regardless of user intent.\n+        # See #26803: daily token quota must fall back like a 402 credit error.\n         is_auto = resolved_provider in {\"auto\", \"\", None}\n-        if should_fallback and is_auto:\n+        is_capacity_error = _is_payment_error(first_err) or _is_connection_error(first_err)\n+        if should_fallback and (is_auto or is_capacity_error):\n             if _is_payment_error(first_err):\n                 reason = \"payment error\"\n                 _mark_provider_unhealthy(","@@ -4871,8 +5047,23 @@ async def async_call_llm(\n                 reason = \"connection error\"\n             logger.info(\"Auxiliary %s (async): %s on %s (%s), trying fallback\",\n                         task or \"call\", reason, resolved_provider, first_err)\n-            fb_client, fb_model, fb_label = _try_payment_fallback(\n-                resolved_provider, task, reason=reason)\n+\n+            # Fallback order (#26882, #26803):\n+            #   1. User-configured fallback_chain (per-task) if set\n+            #   2. Main agent model (last-resort safety net)\n+            # Auto users get the full auto-detection chain instead — its\n+            # Step 1 IS the main agent model.\n+            fb_client, fb_model, fb_label = (None, None, \"\")\n+            if is_auto:\n+                fb_client, fb_model, fb_label = _try_payment_fallback(\n+                    resolved_provider, task, reason=reason)\n+            else:\n+                fb_client, fb_model, fb_label = _try_configured_fallback_chain(\n+                    task, resolved_provider or \"auto\", reason=reason)\n+                if fb_client is None:\n+                    fb_client, fb_model, fb_label = _try_main_agent_model_fallback(\n+                        resolved_provider, task, reason=reason)\n+\n             if fb_client is not None:\n                 fb_kwargs = _build_call_kwargs(\n                     fb_label, fb_model, messages,","@@ -4888,6 +5079,12 @@ async def async_call_llm(\n                     fb_kwargs[\"model\"] = async_fb_model\n                 return _validate_llm_response(\n                     await async_fb.chat.completions.create(**fb_kwargs), task)\n+            # All fallback layers exhausted — warn before re-raising. (#26882)\n+            logger.warning(\n+                \"Auxiliary %s (async): %s on %s and all fallbacks exhausted \"\n+                \"(fallback_chain + main agent model). Raising original error.\",\n+                task or \"call\", reason, resolved_provider,\n+            )\n         # Mirror the sync path: drop poisoned clients on connection/timeout\n         # so the next aux call rebuilds.  See issue #23432.\n         if _is_connection_error(first_err):"]},{"file":"scripts/release.py","hunks":["@@ -184,6 +184,7 @@\n     \"[redacted:email]\": \"npmisantosh\",\n     \"[redacted:email]\": \"luarss\",\n     \"[redacted:email]\": \"zccyman\",\n+    \"[redacted:email]\": \"zccyman\",  # PR #26998 (auxiliary fallback chain)\n     \"[redacted:email]\": \"Osraka\",\n     \"[redacted:email]\": \"Osraka\",\n     \"[redacted:email]\": \"ryptotalent\","]},{"file":"tests/agent/test_auxiliary_client.py","hunks":["@@ -923,6 +923,44 @@ def test_no_status_code_no_message(self):\n         exc = Exception(\"connection reset\")\n         assert _is_payment_error(exc) is False\n \n+    # ── Daily / monthly quota exhaustion (#26803) ────────────────────────────\n+\n+    def test_429_quota_exceeded(self):\n+        \"\"\"Cloud provider quota exhaustion (e.g. Vertex AI) is a payment error.\"\"\"\n+        exc = Exception(\"RESOURCE_EXHAUSTED: quota exceeded for project\")\n+        exc.status_code = 429\n+        assert _is_payment_error(exc) is True\n+\n+    def test_429_too_many_tokens_per_day(self):\n+        \"\"\"Bedrock / LiteLLM daily token limit is a payment error.\"\"\"\n+        exc = Exception(\"Too many tokens per day: 1000000 used, 1000000 limit\")\n+        exc.status_code = 429\n+        assert _is_payment_error(exc) is True\n+\n+    def test_429_daily_limit_phrase(self):\n+        \"\"\"Generic 'daily limit' phrasing is a payment error.\"\"\"\n+        exc = Exception(\"You have exceeded your daily limit.\")\n+        exc.status_code = 429\n+        assert _is_payment_error(exc) is True\n+\n+    def test_429_resource_exhausted_grpc(self):\n+        \"\"\"Vertex AI gRPC RESOURCE_EXHAUSTED maps to payment error.\"\"\"\n+        exc = Exception(\"resource exhausted\")\n+        exc.status_code = 429\n+        assert _is_payment_error(exc) is True\n+\n+    def test_429_daily_quota_phrase(self):\n+        \"\"\"'daily quota' phrasing is a payment error.\"\"\"\n+        exc = Exception(\"Daily quota of 500 requests reached.\")\n+        exc.status_code = 429\n+        assert _is_payment_error(exc) is True\n+\n+    def test_429_transient_rate_limit_not_quota(self):\n+        \"\"\"Transient 429 rate limit without quota keywords is NOT a payment error.\"\"\"\n+        exc = Exception(\"Rate limit exceeded. Retry after 10s.\")\n+        exc.status_code = 429\n+        assert _is_payment_error(exc) is False\n+\n \n class TestIsRateLimitError:\n     \"\"\"_is_rate_limit_error detects 429 rate-limit errors warranting fallback.\"\"\"","@@ -1111,6 +1149,140 @@ def test_429_rate_limit_triggers_fallback(self, monkeypatch):\n         # Fallback client should have been used\n         assert fallback_client.chat.completions.create.called\n \n+\n+class TestAuxiliaryFallbackLayering:\n+    \"\"\"Explicit-provider users get layered fallback: configured_chain → main agent → warn.\"\"\"\n+\n+    def _make_payment_err(self):\n+        exc = Exception(\"Payment Required: insufficient credits\")\n+        exc.status_code = 402\n+        return exc\n+\n+    def test_explicit_provider_uses_configured_chain_first(self, monkeypatch, caplog):\n+        \"\"\"When a user has fallback_chain configured, it's tried BEFORE the main agent model.\"\"\"\n+        monkeypatch.setenv(\"OPENROUTER_API_KEY\", \"or-key\")\n+\n+        primary_client = MagicMock()\n+        primary_client.chat.completions.create.side_effect = self._make_payment_err()\n+\n+        chain_client = MagicMock()\n+        chain_client.chat.completions.create.return_value = MagicMock(choices=[\n+            MagicMock(message=MagicMock(content=\"from configured chain\"))\n+        ])\n+\n+        main_called = MagicMock()\n+\n+        with patch(\"agent.auxiliary_client._get_cached_client\",\n+                   return_value=(primary_client, \"glm-4v-flash\")), \\\n+             patch(\"agent.auxiliary_client._resolve_task_provider_model\",\n+                   return_value=(\"glm\", \"glm-4v-flash\", None, None, None)), \\\n+             patch(\"agent.auxiliary_client._try_configured_fallback_chain\",\n+                   return_value=(chain_client, \"gpt-4o-mini\", \"fallback_chain[0](openai)\")), \\\n+             patch(\"agent.auxiliary_client._try_main_agent_model_fallback\",\n+                   side_effect=main_called):\n+            result = call_llm(\n+                task=\"vision\",\n+                messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n+            )\n+\n+        assert chain_client.chat.completions.create.called\n+        # Main agent fallback should NOT have been consulted — chain succeeded first\n+        main_called.assert_not_called()\n+\n+    def test_explicit_provider_falls_back_to_main_when_chain_exhausted(self, monkeypatch):\n+        \"\"\"If configured fallback_chain returns nothing, main agent model is tried next.\"\"\"\n+        monkeypatch.setenv(\"OPENROUTER_API_KEY\", \"or-key\")\n+\n+        primary_client = MagicMock()\n+        primary_client.chat.completions.create.side_effect = self._make_payment_err()\n+\n+        main_client = MagicMock()\n+        main_client.chat.completions.create.return_value = MagicMock(choices=[\n+            MagicMock(message=MagicMock(content=\"from main agent\"))\n+        ])\n+\n+        with patch(\"agent.auxiliary_client._get_cached_client\",\n+                   return_value=(primary_client, \"glm-4v-flash\")), \\\n+             patch(\"agent.auxiliary_client._resolve_task_provider_model\",\n+                   return_value=(\"glm\", \"glm-4v-flash\", None, None, None)), \\\n+             patch(\"agent.auxiliary_client._try_configured_fallback_chain\",\n+                   return_value=(None, None, \"\")), \\\n+             patch(\"agent.auxiliary_client._try_main_agent_model_fallback\",\n+                   return_value=(main_client, \"claude-sonnet-4\", \"main-agent(openrouter)\")):\n+            result = call_llm(\n+                task=\"vision\",\n+                messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n+            )\n+\n+        assert main_client.chat.completions.create.called\n+\n+    def test_warning_emitted_when_all_fallbacks_exhausted(self, monkeypatch, caplog):\n+        \"\"\"When chain AND main model both fail, a user-visible warning fires before re-raise.\"\"\"\n+        monkeypatch.setenv(\"OPENROUTER_API_KEY\", \"or-key\")\n+\n+        primary_client = MagicMock()\n+        primary_client.chat.completions.create.side_effect = self._make_payment_err()\n+\n+        with patch(\"agent.auxiliary_client._get_cached_client\",\n+                   return_value=(primary_client, \"glm-4v-flash\")), \\\n+             patch(\"agent.auxiliary_client._resolve_task_provider_model\",\n+                   return_value=(\"glm\", \"glm-4v-flash\", None, None, None)), \\\n+             patch(\"agent.auxiliary_client._try_configured_fallback_chain\",\n+                   return_value=(None, None, \"\")), \\\n+             patch(\"agent.auxiliary_client._try_main_agent_model_fallback\",\n+                   return_value=(None, None, \"\")), \\\n+             caplog.at_level(\"WARNING\", logger=\"agent.auxiliary_client\"):\n+            with pytest.raises(Exception, match=\"Payment Required\"):\n+                call_llm(\n+                    task=\"vision\",\n+                    messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n+                )\n+\n+        assert any(\n+            \"all fallbacks exhausted\" in r.message for r in caplog.records\n+        ), f\"Expected exhaustion warning, got: {[r.message for r in caplog.records]}\"\n+\n+\n+class TestTryMainAgentModelFallback:\n+    \"\"\"_try_main_agent_model_fallback resolves the user's main provider+model as a safety net.\"\"\"\n+\n+    def test_returns_none_when_main_provider_is_auto(self):\n+        from agent.auxiliary_client import _try_main_agent_model_fallback\n+        with patch(\"agent.auxiliary_client._read_main_provider\", return_value=\"auto\"), \\\n+             patch(\"agent.auxiliary_client._read_main_model\", return_value=\"some-model\"):\n+            client, model, label = _try_main_agent_model_fallback(\"glm\", task=\"vision\")\n+        assert client is None and model is None and label == \"\"\n+\n+    def test_returns_none_when_failed_provider_equals_main(self):\n+        \"\"\"If the thing that failed IS the main model, no point retrying it.\"\"\"\n+        from agent.auxiliary_client import _try_main_agent_model_fallback\n+        with patch(\"agent.auxiliary_client._read_main_provider\", return_value=\"openrouter\"), \\\n+             patch(\"agent.auxiliary_client._read_main_model\", return_value=\"anthropic/claude-sonnet-4\"):\n+            client, model, label = _try_main_agent_model_fallback(\"openrouter\", task=\"vision\")\n+        assert client is None and label == \"\"\n+\n+    def test_resolves_main_provider_client(self):\n+        from agent.auxiliary_client import _try_main_agent_model_fallback\n+        fake_client = MagicMock()\n+        with patch(\"agent.auxiliary_client._read_main_provider\", return_value=\"openrouter\"), \\\n+             patch(\"agent.auxiliary_client._read_main_model\", return_value=\"anthropic/claude-sonnet-4\"), \\\n+             patch(\"agent.auxiliary_client._is_provider_unhealthy\", return_value=False), \\\n+             patch(\"agent.auxiliary_client.resolve_provider_client\",\n+                   return_value=(fake_client, \"anthropic/claude-sonnet-4\")):\n+            client, model, label = _try_main_agent_model_fallback(\"glm\", task=\"vision\")\n+        assert client is fake_client\n+        assert model == \"anthropic/claude-sonnet-4\"\n+        assert label == \"main-agent(openrouter)\"\n+\n+    def test_skips_when_main_provider_is_unhealthy(self):\n+        from agent.auxiliary_client import _try_main_agent_model_fallback\n+        with patch(\"agent.auxiliary_client._read_main_provider\", return_value=\"openrouter\"), \\\n+             patch(\"agent.auxiliary_client._read_main_model\", return_value=\"anthropic/claude-sonnet-4\"), \\\n+             patch(\"agent.auxiliary_client._is_provider_unhealthy\", return_value=True):\n+            client, model, label = _try_main_agent_model_fallback(\"glm\", task=\"vision\")\n+        assert client is None\n+\n+\n # ---------------------------------------------------------------------------\n # Gate: _resolve_api_key_provider must skip anthropic when not configured\n # ---------------------------------------------------------------------------","@@ -2349,10 +2521,13 @@ def close(self):\n     def test_call_llm_evicts_on_connection_error_with_explicit_provider(self):\n         \"\"\"Connection error on an explicit provider must drop the cached client.\n \n-        This is the exact reporter scenario: ``auxiliary.compression.provider:\n-        main`` (resolves to ``openai-codex``) → no fallback chain runs (not\n-        auto), but the cached client was poisoned by a prior timeout and must\n-        be evicted so the next call rebuilds.\n+        Reporter scenario: ``auxiliary.compression.provider: main`` (resolves\n+        to ``openai-codex``).  After #26803, capacity errors (payment/quota/\n+        connection) DO trigger fallback even on explicit providers — so we\n+        also stub ``_try_payment_fallback`` to ``(None, None, \"\")`` so the\n+        connection error re-raises after eviction instead of escaping into\n+        a real network call.  The contract under test is cache eviction,\n+        not the fallback gate.\n         \"\"\"\n         from agent.auxiliary_client import _client_cache, _client_cache_lock\n ","@@ -2372,6 +2547,9 @@ def test_call_llm_evicts_on_connection_error_with_explicit_provider(self):\n             ), patch(\n                 \"agent.auxiliary_client._get_cached_client\",\n                 return_value=(poisoned, \"gpt-5.5\"),\n+            ), patch(\n+                \"agent.auxiliary_client._try_payment_fallback\",\n+                return_value=(None, None, \"\"),\n             ):\n                 with pytest.raises(ConnectionError):\n                     call_llm(","@@ -2405,6 +2583,9 @@ async def test_async_call_llm_evicts_on_connection_error_with_explicit_provider(\n             ), patch(\n                 \"agent.auxiliary_client._get_cached_client\",\n                 return_value=(poisoned, \"gpt-5.5\"),\n+            ), patch(\n+                \"agent.auxiliary_client._try_payment_fallback\",\n+                return_value=(None, None, \"\"),\n             ):\n                 with pytest.raises(ConnectionError):\n                     await async_call_llm("]},{"file":"website/docs/user-guide/features/fallback-providers.md","hunks":["@@ -320,6 +320,55 @@ auxiliary:\n \n ---\n \n+## Auxiliary Capacity-Error Fallback\n+\n+When you set an explicit auxiliary provider (e.g. `auxiliary.vision.provider: glm`), Hermes treats that as your preferred choice — but if the provider literally cannot serve the request because of a **capacity error** (HTTP 402 payment required, HTTP 429 daily-quota exhaustion, connection failure), Hermes falls back through a layered chain instead of failing silently:\n+\n+1. **Primary aux provider** — the one you configured (tried first, always)\n+2. **`auxiliary.<task>.fallback_chain`** — your per-task override list, if you wrote one\n+3. **Main agent provider + model** — last-resort safety net (always tried, even if you didn't write a chain)\n+4. **Warn + re-raise** — if every layer fails, Hermes logs `Auxiliary <task>: ... all fallbacks exhausted` at WARNING level and re-raises the original error\n+\n+Transient HTTP 429 rate limits (`Retry-After: ...`) are treated as request constraints, not capacity problems — they respect your explicit provider choice and do **not** trigger the fallback ladder. Only daily/monthly quota exhaustion, payment errors, and connection failures bypass the explicit-provider gate.\n+\n+For users on `provider: auto` (no explicit aux provider), the existing auto-detection chain runs in place of steps 2–3. Its first step is already the main agent model, so `auto` users get the same outcome with zero config.\n+\n+### Optional: per-task fallback chain\n+\n+If you want a different fallback ordering than \"main agent model first\", configure `fallback_chain` explicitly. Each entry needs at least `provider`; `model`, `base_url`, and `api_key` are optional.\n+\n+```yaml\n+auxiliary:\n+  vision:\n+    provider: glm\n+    model: glm-4v-flash\n+    fallback_chain:\n+      - provider: openrouter\n+        model: google/gemini-3-flash-preview\n+      - provider: nous\n+        model: anthropic/claude-sonnet-4\n+\n+  compression:\n+    provider: openrouter\n+    fallback_chain:\n+      - provider: openai\n+        model: gpt-4o-mini\n+```\n+\n+You do **not** need to configure `fallback_chain` to get fallback — the main-agent safety net runs regardless. Use it only when you specifically want a different order than the default.\n+\n+### Provider quota errors that trigger fallback\n+\n+Hermes recognizes these as capacity-equivalent to 402 credit exhaustion (not transient rate limits):\n+\n+- Bedrock / LiteLLM: `Too many tokens per day`, `daily limit`, `tokens per day`\n+- Vertex AI / GCP: `quota exceeded`, `resource exhausted`, `RESOURCE_EXHAUSTED`\n+- Generic: `daily quota`, `quota_exceeded`\n+\n+If your provider returns a different phrase for daily-quota exhaustion and Hermes doesn't trigger fallback, that's a bug — open an issue with the exact error string.\n+\n+---\n+\n ## Context Compression Fallback\n \n Context compression uses the `auxiliary.compression` config block to control which model and provider handles summarization:","@@ -378,14 +427,16 @@ See [Scheduled Tasks (Cron)](/docs/user-guide/features/cron) for full configurat\n | Feature | Fallback Mechanism | Config Location |\n |---------|-------------------|----------------|\n | Main agent model | `fallback_model` in config.yaml — per-turn failover on errors (primary restored each turn) | `fallback_model:` (top-level) |\n-| Vision | Auto-detection chain + internal OpenRouter retry | `auxiliary.vision` |\n-| Web extraction | Auto-detection chain + internal OpenRouter retry | `auxiliary.web_extract` |\n-| Context compression | Auto-detection chain, degrades to no-summary if unavailable | `auxiliary.compression` |\n-| Session search | Auto-detection chain | `auxiliary.session_search` |\n-| Skills hub | Auto-detection chain | `auxiliary.skills_hub` |\n-| MCP helpers | Auto-detection chain | `auxiliary.mcp` |\n-| Approval classification | Auto-detection chain | `auxiliary.approval` |\n-| Title generation | Auto-detection chain | `auxiliary.title_generation` |\n-| Triage specifier | Auto-detection chain | `auxiliary.triage_specifier` |\n+| Auxiliary tasks (any) — auto users | Full auto-detection chain (main agent model first, then provider chain) on capacity errors | `auxiliary.<task>.provider: auto` |\n+| Auxiliary tasks (any) — explicit provider | `fallback_chain` (if set) → main agent model → warn + raise, on capacity errors only | `auxiliary.<task>.fallback_chain` |\n+| Vision | Layered (see above) + internal OpenRouter retry | `auxiliary.vision` |\n+| Web extraction | Layered (see above) + internal OpenRouter retry | `auxiliary.web_extract` |\n+| Context compression | Layered (see above); degrades to no-summary if all layers unavailable | `auxiliary.compression` |\n+| Session search | Layered (see above) | `auxiliary.session_search` |\n+| Skills hub | Layered (see above) | `auxiliary.skills_hub` |\n+| MCP helpers | Layered (see above) | `auxiliary.mcp` |\n+| Approval classification | Layered (see above) | `auxiliary.approval` |\n+| Title generation | Layered (see above) | `auxiliary.title_generation` |\n+| Triage specifier | Layered (see above) | `auxiliary.triage_specifier` |\n | Delegation | Provider override only (no automatic fallback) | `delegation.provider` / `delegation.model` |\n | Cron jobs | Per-job provider override only (no automatic fallback) | Per-job `provider` / `model` |\n"]}]},{"kind":"pr","number":32411,"title":"feat(auxiliary): add per-task fallback_providers config for transparent failover","url":null,"state":"open","close_reason":{"refs":[{"kind":"pr","value":"#32411","number":32411},{"kind":"pr","value":"#31168","number":31168},{"kind":"pr","value":"#31168","number":31168},{"kind":"pr","value":"#31168","number":31168},{"kind":"pr","value":"#31168","number":31168}],"category":"superseded_by_pr","implication":"superseded"},"merged":false},{"kind":"pr","number":129725,"title":"feat(tts): opt-in provider fallback chain (tts.fallback_chain)","url":null,"state":"open","close_reason":{"category":"uncategorized","implication":"needs_llm"},"merged":false}],"trust":{"default_tier":"verified","includes_suggested":false}}