# NousResearch/hermes-agent · Fallback chain fixes for rate limits and errors ## Open now - 3 issues, 5 PRs open on GitHub. - #25822: auxiliary vision: Gemini 503 (UNAVAILABLE) does not trigger provider fallback - #50995: Vision fallback silently ignored when configured in legacy singular form - #51513: fix(vision): fallback chain completely non-functional — 5 bugs, complete fix with verification - PR #25843: fix(auxiliary): trigger provider fallback on 5xx server errors (#25822) - PR #26152: fix(auxiliary): fall back to alternative provider on 5xx server errors - PR #36280: fix(agent): trigger fallback on quota-exhaustion 429s (e.g. Kimi monthly limit) - PR #51002: fix(auxiliary): accept legacy fallback_provider/fallback_model in auxiliary task fallback chain - PR #57115: fix(vision): make fallback chain iterate all entries instead of stopping at first These issues and PRs address multiple bugs in fallback logic across agent, auxiliary, and vision components, focusing on cases where 429 rate-limit/quota-exhaustion errors and 5xx server errors fail to trigger provider fallback chains. Several entries also address legacy configuration formats not being recognized by fallback logic. ## TL;DR - 27 member(s): 3 open issue(s), 5 closed issue(s), 5 open PR(s), 3 merged PR(s), 11 PR(s) closed without merging. - A fix was merged to main in PR #27625. Verify it addresses your case. - **Recommended:** read PR #27625 (and any competing PRs below) before filing. Confirm the merged change covers your symptom. ## Maintainer resolution verdict - Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-27625`. This is distinct from the fix status above. Weigh it yourself. ## How this cluster was grouped - Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text. - Members outside that structural core may belong to a different underlying problem. - Pairwise link density `0.06`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked. ## Open related issues - #25822: auxiliary vision: Gemini 503 (UNAVAILABLE) does not trigger provider fallback - #50995: Vision fallback silently ignored when configured in legacy singular form - #51513: fix(vision): fallback chain completely non-functional — 5 bugs, complete fix with verification ## Closed / duplicate issues - #26803: Auxiliary call_llm fallback doesn't trigger on provider rate limits (429 daily quota) - #36276: fix(agent): HTTP 429 quota-exhaustion errors retried 3 times instead of failing fast - #49586: bug(vision): fallback_chain crash — 'await' on sync OpenAI client makes fallback path non-functional - #52228: Auxiliary rate-limit fallback bypassed for explicit providers - #65563: Ollama Cloud "session usage limit" 429 error not triggering fallback_providers chain ## Prior & competing PRs - **#26002: fix: add server error (5xx) detection to auxiliary vision fallback**. closed without merge · fix tier `suggested` - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#25843`, `#25843` - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._ - **#27625: feat(auxiliary): layered fallback (chain → main agent) + capacity-error gate fix**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `agent/auxiliary_client.py`, `scripts/release.py`, `tests/agent/test_auxiliary_client.py`, `website/docs/user-guide/features/fallback-providers.md` - fix hunks: - `agent/auxiliary_client.py` ```diff @@ -2087,7 +2087,13 @@ def _is_payment_error(exc: Exception) -> bool: """Detect payment/credit/quota exhaustion errors. Returns True for HTTP 402 (Payment Required) and for 429/other errors - whose message indicates billing exhaustion rather than rate limiting. + whose message indicates billing exhaustion or daily quota exhaustion + rather than transient rate limiting. + + Daily token quota errors (e.g. Bedrock "Too many tokens per day", + Vertex AI "quota exceeded") are functionally equivalent to credit + exhaustion — the provider cannot serve the request until the quota + resets — and should trigger the same provider-fallback logic. """ status = getattr(exc, "status_code", None) if status == 402: ``` ```diff @@ -2095,10 +2101,19 @@ def _is_payment_error(exc: Exception) -> bool: err_lower = str(exc).lower() # OpenRouter and other providers include "credits" or "afford" in 402 bodies, # but sometimes wrap them in 429 or other codes. + # Daily quota exhaustion from Bedrock, Vertex AI, and similar providers + # uses different language but is semantically identical to credit exhaustion. if status in {402, 429, None}: - if any(kw in err_lower for kw in ("credits", "insufficient funds", - "can only afford", "billing", - "payment required")): + if any(kw in err_lower for kw in ( + "credits", "insufficient funds", + "can only afford", "billing", + "payment required", + # Daily / monthly quota exhaustion keywords + "quota exceeded", "quota_exceeded", + "too many tokens per day", "daily limit", + "tokens per day", "daily quota", + "resource exhausted", # Vertex AI / gRPC quota errors + )): return True return False ``` ```diff @@ -2579,6 +2594,133 @@ def _try_payment_fallback( return None, None, "" +def _try_main_agent_model_fallback( + failed_provider: str, + task: str = None, + reason: str = "error", +) -> Tuple[Optional[Any], Optional[str], str]: + """Last-resort fallback to the user's main agent provider + model. + + Used after the configured fallback_chain is exhausted (or empty) for + users with an explicit auxiliary provider. This is the "safety net" + layer: if nothing the user asked for can serve the request, try the + main chat model before giving up. + + Skips when the failed provider already IS the main provider (no point + retrying the same backend that just failed). + + Returns: + (client, model, provider_label) or (None, None, "") if no fallback. + """ + main_provider = (_read_main_provider() or "").strip() + main_model = (_read_main_model() or "").strip() + if not main_provider or not main_model or main_provider.lower() in {"auto", ""}: + return None, None, "" + + skip = (failed_provider or "").lower().strip() + if main_provider.lower() == skip: + # The thing that failed IS the main model — nothing to fall back to. + return None, None, "" + if _is_provider_unhealthy(main_provider): + _log_skip_unhealthy(main_provider, task) + return None, None, "" + + try: + client, resolved_model = resolve_provider_client( + provider=main_provider, model=main_model, + ) + except Exception: + client, resolved_model = None, None + + if client is None: + return None, None, "" + + label = f"main-agent({main_provider})" + logger.info( + "Auxiliary %s: %s on %s — falling back to main agent model %s (%s)", + task or "call", reason, failed_provider, label, resolved_model or main_model, + ) + return client, resolved_model or main_model, label + + +def _try_configured_fallback_chain( + task: str, + failed_provider: str, + reason: str = "error", +) -> Tuple[Optional[Any], Optional[str], str]: + """Try user-configured fallback_chain for a specific auxiliary task. + + Reads auxiliary..fallback_chain from config.yaml and tries each + entry in order. Each entry must have at least ``provider``; ``model``, + ``base_url``, and ``api_key`` are optional. + + Returns: + (client, model, provider_label) or (None, None, "") if no fallback. + """ + if not task: + return None, None, "" + + task_config = _get_auxiliary_task_config(task) + chain = task_config.get("fallback_chain") + if not chain or not isinstance(chain, list): + return None, None, "" + + skip = failed_provider.lower().strip() + tried = [] + + for i, entry in enumerate(chain): + if not isinstance(entry, dict): + continue + fb_provider = str(entry.get("provider", "")).strip() + if not fb_provider or fb_provider.lower() == skip: + continue + fb_model = str(entry.get("model", "")).strip() or None + fb_base_url = str(entry.get("base_url", "")).strip() or None + fb_api_key = [redacted:secret-assignment]"api_key", "")).strip() or None + + label = f"fallback_chain[{i}]({fb_provider})" + + try: + fb_client = _resolve_single_provider( + fb_provider, fb_model, fb_base_url, fb_api_key) + except Exception: + fb_client = None + + if fb_client is not None: + logger.info( + "Auxiliary %s: %s on %s — configured fallback to %s (%s)", + task, reason, failed_provider, label, fb_model or "default", + ) + return fb_client, fb_model, label + tried.append(label) + + if tried: + logger.debug( + "Auxiliary %s: configured fallback_chain exhausted (tried: %s)", + task, ", ".join(tried), + ) + return None, None, "" + + +def _resolve_single_provider( + provider: str, + model: Optional[str] = None, + base_url: Optional[str] = None, + api_key: [redacted:secret-assignment] = None, +) -> Optional[Any]: + """Resolve a single provider entry from fallback_chain to an OpenAI client. + + Uses the existing provider resolution infrastructure where possible. + """ + # Reuse resolve_provider_client which handles provider→client mapping + client, resolved_model = resolve_provider_client( + provider=provider, + model=model, + base_url=base_url, + api_key=[redacted:secret-assignment] + ) + return client + def _resolve_auto(main_runtime: Optional[Dict[str, Any]] = None) -> Tuple[Optional[OpenAI], Optional[str]]: """Full auto-detection chain. ``` ```diff @@ -4526,11 +4668,17 @@ def call_llm( or _is_connection_error(first_err) or _is_rate_limit_error(first_err) ) - # Only try alternative providers when the user didn't explicitly - # configure this task's provider. Explicit provider = hard constraint; - # auto (the default) = best-effort fallback chain. (#7559) + # Respect explicit provider choice for transient errors (auth, request + # validation, etc.) but allow fallback when the provider clearly cannot + # serve the request due to capacity: payment/quota exhaustion and + # connection failures are capacity problems, not request constraints. + # See #26803: daily token quota (429 + "too many tokens per day") must + # fall back just like a 402 credit error. is_auto = resolved_provider in {"auto", "", None} - if should_fallback and is_auto: + # Capacity errors bypass the explicit-provider gate: the provider + # literally cannot serve this request regardless of user intent. + is_capacity_error = _is_payment_error(first_err) or _is_connection_error(first_err) + if should_fallback and (is_auto or is_capacity_error): if _is_payment_error(first_err): reason = "payment error" # Resolve the actual provider label (resolved_provider may be ``` ```diff @@ -4546,8 +4694,24 @@ def call_llm( reason = "connection error" logger.info("Auxiliary %s: %s on %s (%s), trying fallback", task or "call", reason, resolved_provider, first_err) - fb_client, fb_model, fb_label = _try_payment_fallback( - resolved_provider, task, reason=reason) + + # Fallback order (#26882, #26803): + # 1. User-configured fallback_chain (per-task) if set + # 2. Main agent model (last-resort safety net) + # For auto users (no explicit aux provider), use the full + # auto-detection chain instead — its Step 1 IS the main agent + # model, so users on `auto` already get main-model fallback. + fb_client, fb_model, fb_label = (None, None, "") + if is_auto: + fb_client, fb_model, fb_label = _try_payment_fallback( + resolved_provider, task, reason=reason) + else: + fb_client, fb_model, fb_label = _try_configured_fallback_chain( + task, resolved_provider or "auto", reason=reason) + if fb_client is None: + fb_client, fb_model, fb_label = _try_main_agent_model_fallback( + resolved_provider, task, reason=reason) + if fb_client is not None: fb_kwargs = _build_call_kwargs( fb_label, fb_model, messages, ``` ```diff @@ -4557,6 +4721,14 @@ def call_llm( base_url=str(getattr(fb_client, "base_url", "") or "")) return _validate_llm_response( fb_client.chat.completions.create(**fb_kwargs), task) + # All fallback layers exhausted — emit a single user-visible + # warning so the operator knows aux task is about to fail. + # (#26882) The error itself is re-raised below. + logger.warning( + "Auxiliary %s: %s on %s and all fallbacks exhausted " + "(fallback_chain + main agent model). Raising original error.", + task or "call", reason, resolved_provider, + ) # Connection/timeout errors leave the cached client poisoned (closed # httpx transport, half-read stream, dead async loop). Drop it from # the cache regardless of whether we found a fallback above so the ``` ```diff @@ -4858,8 +5030,12 @@ async def async_call_llm( or _is_connection_error(first_err) or _is_rate_limit_error(first_err) ) + # Capacity errors (payment/quota/connection) bypass the explicit-provider + # gate — the provider cannot serve the request regardless of user intent. + # See #26803: daily token quota must fall back like a 402 credit error. is_auto = resolved_provider in {"auto", "", None} - if should_fallback and is_auto: + is_capacity_error = _is_payment_error(first_err) or _is_connection_error(first_err) + if should_fallback and (is_auto or is_capacity_error): if _is_payment_error(first_err): reason = "payment error" _mark_provider_unhealthy( ``` ```diff @@ -4871,8 +5047,23 @@ async def async_call_llm( reason = "connection error" logger.info("Auxiliary %s (async): %s on %s (%s), trying fallback", task or "call", reason, resolved_provider, first_err) - fb_client, fb_model, fb_label = _try_payment_fallback( - resolved_provider, task, reason=reason) + + # Fallback order (#26882, #26803): + # 1. User-configured fallback_chain (per-task) if set + # 2. Main agent model (last-resort safety net) + # Auto users get the full auto-detection chain instead — its + # Step 1 IS the main agent model. + fb_client, fb_model, fb_label = (None, None, "") + if is_auto: + fb_client, fb_model, fb_label = _try_payment_fallback( + resolved_provider, task, reason=reason) + else: + fb_client, fb_model, fb_label = _try_configured_fallback_chain( + task, resolved_provider or "auto", reason=reason) + if fb_client is None: + fb_client, fb_model, fb_label = _try_main_agent_model_fallback( + resolved_provider, task, reason=reason) + if fb_client is not None: fb_kwargs = _build_call_kwargs( fb_label, fb_model, messages, ``` ```diff @@ -4888,6 +5079,12 @@ async def async_call_llm( fb_kwargs["model"] = async_fb_model return _validate_llm_response( await async_fb.chat.completions.create(**fb_kwargs), task) + # All fallback layers exhausted — warn before re-raising. (#26882) + logger.warning( + "Auxiliary %s (async): %s on %s and all fallbacks exhausted " + "(fallback_chain + main agent model). Raising original error.", + task or "call", reason, resolved_provider, + ) # Mirror the sync path: drop poisoned clients on connection/timeout # so the next aux call rebuilds. See issue #23432. if _is_connection_error(first_err): ``` - `scripts/release.py` ```diff @@ -184,6 +184,7 @@ "[redacted:email]": "npmisantosh", "[redacted:email]": "luarss", "[redacted:email]": "zccyman", + "[redacted:email]": "zccyman", # PR #26998 (auxiliary fallback chain) "[redacted:email]": "Osraka", "[redacted:email]": "Osraka", "[redacted:email]": "ryptotalent", ``` - `tests/agent/test_auxiliary_client.py` ```diff @@ -923,6 +923,44 @@ def test_no_status_code_no_message(self): exc = Exception("connection reset") assert _is_payment_error(exc) is False + # ── Daily / monthly quota exhaustion (#26803) ──────────────────────────── + + def test_429_quota_exceeded(self): + """Cloud provider quota exhaustion (e.g. Vertex AI) is a payment error.""" + exc = Exception("RESOURCE_EXHAUSTED: quota exceeded for project") + exc.status_code = 429 + assert _is_payment_error(exc) is True + + def test_429_too_many_tokens_per_day(self): + """Bedrock / LiteLLM daily token limit is a payment error.""" + exc = Exception("Too many tokens per day: 1000000 used, 1000000 limit") + exc.status_code = 429 + assert _is_payment_error(exc) is True + + def test_429_daily_limit_phrase(self): + """Generic 'daily limit' phrasing is a payment error.""" + exc = Exception("You have exceeded your daily limit.") + exc.status_code = 429 + assert _is_payment_error(exc) is True + + def test_429_resource_exhausted_grpc(self): + """Vertex AI gRPC RESOURCE_EXHAUSTED maps to payment error.""" + exc = Exception("resource exhausted") + exc.status_code = 429 + assert _is_payment_error(exc) is True + + def test_429_daily_quota_phrase(self): + """'daily quota' phrasing is a payment error.""" + exc = Exception("Daily quota of 500 requests reached.") + exc.status_code = 429 + assert _is_payment_error(exc) is True + + def test_429_transient_rate_limit_not_quota(self): + """Transient 429 rate limit without quota keywords is NOT a payment error.""" + exc = Exception("Rate limit exceeded. Retry after 10s.") + exc.status_code = 429 + assert _is_payment_error(exc) is False + class TestIsRateLimitError: """_is_rate_limit_error detects 429 rate-limit errors warranting fallback.""" ``` ```diff @@ -1111,6 +1149,140 @@ def test_429_rate_limit_triggers_fallback(self, monkeypatch): # Fallback client should have been used assert fallback_client.chat.completions.create.called + +class TestAuxiliaryFallbackLayering: + """Explicit-provider users get layered fallback: configured_chain → main agent → warn.""" + + def _make_payment_err(self): + exc = Exception("Payment Required: insufficient credits") + exc.status_code = 402 + return exc + + def test_explicit_provider_uses_configured_chain_first(self, monkeypatch, caplog): + """When a user has fallback_chain configured, it's tried BEFORE the main agent model.""" + monkeypatch.setenv("OPENROUTER_API_KEY", "or-key") + + primary_client = MagicMock() + primary_client.chat.completions.create.side_effect = self._make_payment_err() + + chain_client = MagicMock() + chain_client.chat.completions.create.return_value = MagicMock(choices=[ + MagicMock(message=MagicMock(content="from configured chain")) + ]) + + main_called = MagicMock() + + with patch("agent.auxiliary_client._get_cached_client", + return_value=(primary_client, "glm-4v-flash")), \ + patch("agent.auxiliary_client._resolve_task_provider_model", + return_value=("glm", "glm-4v-flash", None, None, None)), \ + patch("agent.auxiliary_client._try_configured_fallback_chain", + return_value=(chain_client, "gpt-4o-mini", "fallback_chain[0](openai)")), \ + patch("agent.auxiliary_client._try_main_agent_model_fallback", + side_effect=main_called): + result = call_llm( + task="vision", + messages=[{"role": "user", "content": "hello"}], + ) + + assert chain_client.chat.completions.create.called + # Main agent fallback should NOT have been consulted — chain succeeded first + main_called.assert_not_called() + + def test_explicit_provider_falls_back_to_main_when_chain_exhausted(self, monkeypatch): + """If configured fallback_chain returns nothing, main agent model is tried next.""" + monkeypatch.setenv("OPENROUTER_API_KEY", "or-key") + + primary_client = MagicMock() + primary_client.chat.completions.create.side_effect = self._make_payment_err() + + main_client = MagicMock() + main_client.chat.completions.create.return_value = MagicMock(choices=[ + MagicMock(message=MagicMock(content="from main agent")) + ]) + + with patch("agent.auxiliary_client._get_cached_client", + return_value=(primary_client, "glm-4v-flash")), \ + patch("agent.auxiliary_client._resolve_task_provider_model", + return_value=("glm", "glm-4v-flash", None, None, None)), \ + patch("agent.auxiliary_client._try_configured_fallback_chain", + return_value=(None, None, "")), \ + patch("agent.auxiliary_client._try_main_agent_model_fallback", + return_value=(main_client, "claude-sonnet-4", "main-agent(openrouter)")): + result = call_llm( + task="vision", + messages=[{"role": "user", "content": "hello"}], + ) + + assert main_client.chat.completions.create.called + + def test_warning_emitted_when_all_fallbacks_exhausted(self, monkeypatch, caplog): + """When chain AND main model both fail, a user-visible warning fires before re-raise.""" + monkeypatch.setenv("OPENROUTER_API_KEY", "or-key") + + primary_client = MagicMock() + primary_client.chat.completions.create.side_effect = self._make_payment_err() + + with patch("agent.auxiliary_client._get_cached_client", + return_value=(primary_client, "glm-4v-flash")), \ + patch("agent.auxiliary_client._resolve_task_provider_model", + return_value=("glm", "glm-4v-flash", None, None, None)), \ + patch("agent.auxiliary_client._try_configured_fallback_chain", + return_value=(None, None, "")), \ + patch("agent.auxiliary_client._try_main_agent_model_fallback", + return_value=(None, None, "")), \ + caplog.at_level("WARNING", logger="agent.auxiliary_client"): + with pytest.raises(Exception, match="Payment Required"): + call_llm( + task="vision", + messages=[{"role": "user", "content": "hello"}], + ) + + assert any( + "all fallbacks exhausted" in r.message for r in caplog.records + ), f"Expected exhaustion warning, got: {[r.message for r in caplog.records]}" + + +class TestTryMainAgentModelFallback: + """_try_main_agent_model_fallback resolves the user's main provider+model as a safety net.""" + + def test_returns_none_when_main_provider_is_auto(self): + from agent.auxiliary_client import _try_main_agent_model_fallback + with patch("agent.auxiliary_client._read_main_provider", return_value="auto"), \ + patch("agent.auxiliary_client._read_main_model", return_value="some-model"): + client, model, label = _try_main_agent_model_fallback("glm", task="vision") + assert client is None and model is None and label == "" + + def test_returns_none_when_failed_provider_equals_main(self): + """If the thing that failed IS the main model, no point retrying it.""" + from agent.auxiliary_client import _try_main_agent_model_fallback + with patch("agent.auxiliary_client._read_main_provider", return_value="openrouter"), \ + patch("agent.auxiliary_client._read_main_model", return_value="anthropic/claude-sonnet-4"): + client, model, label = _try_main_agent_model_fallback("openrouter", task="vision") + assert client is None and label == "" + + def test_resolves_main_provider_client(self): + from agent.auxiliary_client import _try_main_agent_model_fallback + fake_client = MagicMock() + with patch("agent.auxiliary_client._read_main_provider", return_value="openrouter"), \ + patch("agent.auxiliary_client._read_main_model", return_value="anthropic/claude-sonnet-4"), \ + patch("agent.auxiliary_client._is_provider_unhealthy", return_value=False), \ + patch("agent.auxiliary_client.resolve_provider_client", + return_value=(fake_client, "anthropic/claude-sonnet-4")): + client, model, label = _try_main_agent_model_fallback("glm", task="vision") + assert client is fake_client + assert model == "anthropic/claude-sonnet-4" + assert label == "main-agent(openrouter)" + + def test_skips_when_main_provider_is_unhealthy(self): + from agent.auxiliary_client import _try_main_agent_model_fallback + with patch("agent.auxiliary_client._read_main_provider", return_value="openrouter"), \ + patch("agent.auxiliary_client._read_main_model", return_value="anthropic/claude-sonnet-4"), \ + patch("agent.auxiliary_client._is_provider_unhealthy", return_value=True): + client, model, label = _try_main_agent_model_fallback("glm", task="vision") + assert client is None + + # --------------------------------------------------------------------------- # Gate: _resolve_api_key_provider must skip anthropic when not configured # --------------------------------------------------------------------------- ``` ```diff @@ -2349,10 +2521,13 @@ def close(self): def test_call_llm_evicts_on_connection_error_with_explicit_provider(self): """Connection error on an explicit provider must drop the cached client. - This is the exact reporter scenario: ``auxiliary.compression.provider: - main`` (resolves to ``openai-codex``) → no fallback chain runs (not - auto), but the cached client was poisoned by a prior timeout and must - be evicted so the next call rebuilds. + Reporter scenario: ``auxiliary.compression.provider: main`` (resolves + to ``openai-codex``). After #26803, capacity errors (payment/quota/ + connection) DO trigger fallback even on explicit providers — so we + also stub ``_try_payment_fallback`` to ``(None, None, "")`` so the + connection error re-raises after eviction instead of escaping into + a real network call. The contract under test is cache eviction, + not the fallback gate. """ from agent.auxiliary_client import _client_cache, _client_cache_lock ``` ```diff @@ -2372,6 +2547,9 @@ def test_call_llm_evicts_on_connection_error_with_explicit_provider(self): ), patch( "agent.auxiliary_client._get_cached_client", return_value=(poisoned, "gpt-5.5"), + ), patch( + "agent.auxiliary_client._try_payment_fallback", + return_value=(None, None, ""), ): with pytest.raises(ConnectionError): call_llm( ``` ```diff @@ -2405,6 +2583,9 @@ async def test_async_call_llm_evicts_on_connection_error_with_explicit_provider( ), patch( "agent.auxiliary_client._get_cached_client", return_value=(poisoned, "gpt-5.5"), + ), patch( + "agent.auxiliary_client._try_payment_fallback", + return_value=(None, None, ""), ): with pytest.raises(ConnectionError): await async_call_llm( ``` - `website/docs/user-guide/features/fallback-providers.md` ```diff @@ -320,6 +320,55 @@ auxiliary: --- +## Auxiliary Capacity-Error Fallback + +When you set an explicit auxiliary provider (e.g. `auxiliary.vision.provider: glm`), Hermes treats that as your preferred choice — but if the provider literally cannot serve the request because of a **capacity error** (HTTP 402 payment required, HTTP 429 daily-quota exhaustion, connection failure), Hermes falls back through a layered chain instead of failing silently: + +1. **Primary aux provider** — the one you configured (tried first, always) +2. **`auxiliary..fallback_chain`** — your per-task override list, if you wrote one +3. **Main agent provider + model** — last-resort safety net (always tried, even if you didn't write a chain) +4. **Warn + re-raise** — if every layer fails, Hermes logs `Auxiliary : ... all fallbacks exhausted` at WARNING level and re-raises the original error + +Transient HTTP 429 rate limits (`Retry-After: ...`) are treated as request constraints, not capacity problems — they respect your explicit provider choice and do **not** trigger the fallback ladder. Only daily/monthly quota exhaustion, payment errors, and connection failures bypass the explicit-provider gate. + +For users on `provider: auto` (no explicit aux provider), the existing auto-detection chain runs in place of steps 2–3. Its first step is already the main agent model, so `auto` users get the same outcome with zero config. + +### Optional: per-task fallback chain + +If you want a different fallback ordering than "main agent model first", configure `fallback_chain` explicitly. Each entry needs at least `provider`; `model`, `base_url`, and `api_key` are optional. + +```yaml +auxiliary: + vision: + provider: glm + model: glm-4v-flash + fallback_chain: + - provider: openrouter + model: google/gemini-3-flash-preview + - provider: nous + model: anthropic/claude-sonnet-4 + + compression: + provider: openrouter + fallback_chain: + - provider: openai + model: gpt-4o-mini +``` + +You do **not** need to configure `fallback_chain` to get fallback — the main-agent safety net runs regardless. Use it only when you specifically want a different order than the default. + +### Provider quota errors that trigger fallback + +Hermes recognizes these as capacity-equivalent to 402 credit exhaustion (not transient rate limits): + +- Bedrock / LiteLLM: `Too many tokens per day`, `daily limit`, `tokens per day` +- Vertex AI / GCP: `quota exceeded`, `resource exhausted`, `RESOURCE_EXHAUSTED` +- Generic: `daily quota`, `quota_exceeded` + +If your provider returns a different phrase for daily-quota exhaustion and Hermes doesn't trigger fallback, that's a bug — open an issue with the exact error string. + +--- + ## Context Compression Fallback Context compression uses the `auxiliary.compression` config block to control which model and provider handles summarization: ``` ```diff @@ -378,14 +427,16 @@ See [Scheduled Tasks (Cron)](/docs/user-guide/features/cron) for full configurat | Feature | Fallback Mechanism | Config Location | |---------|-------------------|----------------| | Main agent model | `fallback_model` in config.yaml — per-turn failover on errors (primary restored each turn) | `fallback_model:` (top-level) | -| Vision | Auto-detection chain + internal OpenRouter retry | `auxiliary.vision` | -| Web extraction | Auto-detection chain + internal OpenRouter retry | `auxiliary.web_extract` | -| Context compression | Auto-detection chain, degrades to no-summary if unavailable | `auxiliary.compression` | -| Session search | Auto-detection chain | `auxiliary.session_search` | -| Skills hub | Auto-detection chain | `auxiliary.skills_hub` | -| MCP helpers | Auto-detection chain | `auxiliary.mcp` | -| Approval classification | Auto-detection chain | `auxiliary.approval` | -| Title generation | Auto-detection chain | `auxiliary.title_generation` | -| Triage specifier | Auto-detection chain | `auxiliary.triage_specifier` | +| Auxiliary tasks (any) — auto users | Full auto-detection chain (main agent model first, then provider chain) on capacity errors | `auxiliary..provider: auto` | +| Auxiliary tasks (any) — explicit provider | `fallback_chain` (if set) → main agent model → warn + raise, on capacity errors only | `auxiliary..fallback_chain` | +| Vision | Layered (see above) + internal OpenRouter retry | `auxiliary.vision` | +| Web extraction | Layered (see above) + internal OpenRouter retry | `auxiliary.web_extract` | +| Context compression | Layered (see above); degrades to no-summary if all layers unavailable | `auxiliary.compression` | +| Session search | Layered (see above) | `auxiliary.session_search` | +| Skills hub | Layered (see above) | `auxiliary.skills_hub` | +| MCP helpers | Layered (see above) | `auxiliary.mcp` | +| Approval classification | Layered (see above) | `auxiliary.approval` | +| Title generation | Layered (see above) | `auxiliary.title_generation` | +| Triage specifier | Layered (see above) | `auxiliary.triage_specifier` | | Delegation | Provider override only (no automatic fallback) | `delegation.provider` / `delegation.model` | | Cron jobs | Per-job provider override only (no automatic fallback) | Per-job `provider` / `model` | ``` - **#49589: fix(vision): fail-proof fallback chain — survive quota limits, VPN glitches, and provider outages**. closed without merge · fix tier `suggested` - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `a57424683759617040dd82082d85128deb236de4`, `#51513` - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._ - **#52456: fix(auxiliary): fall back on rate limits for explicit providers**. closed without merge · fix tier `suggested` - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#52228`, `#52230`, `#52251`, `#52230`, `#52230` - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._ - **#52661: fix(auxiliary): make compression survive a bad/limited fallback (salvage #49076 #52230 #51835 #52421)**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `agent/auxiliary_client.py`, `agent/conversation_compression.py`, `scripts/release.py`, `tests/agent/test_auxiliary_client.py`, `tests/run_agent/test_compression_feasibility.py` - fix hunks: - `agent/auxiliary_client.py` ```diff @@ -101,6 +101,7 @@ def __repr__(self): OpenAI = _OpenAIProxy() # module-level name, resolves lazily on call/isinstance from agent.credential_pool import load_pool +from agent.model_metadata import MINIMUM_CONTEXT_LENGTH, get_model_context_length from hermes_cli.config import get_hermes_home from hermes_constants import OPENROUTER_BASE_URL from utils import base_url_host_matches, base_url_hostname, env_float, model_forces_max_completion_tokens, normalize_proxy_env_vars ``` ```diff @@ -2470,7 +2471,7 @@ def _is_payment_error(exc: Exception) -> bool: # but sometimes wrap them in 429 or other codes. # Daily quota exhaustion from Bedrock, Vertex AI, and similar providers # uses different language but is semantically identical to credit exhaustion. - if status in {402, 404, 429, None}: + if status in {402, 403, 404, 429, None}: if any(kw in err_lower for kw in ( "credits", "insufficient funds", "can only afford", "billing", ``` ```diff @@ -2479,6 +2480,8 @@ def _is_payment_error(exc: Exception) -> bool: "balance_depleted", "no usable credits", "model_not_supported_on_free_tier", "not available on the free tier", + "requires a subscription", "upgrade for access", + "upgrade for higher limits", "reached your session usage limit", # Daily / monthly / weekly quota exhaustion keywords "quota exceeded", "quota_exceeded", "too many tokens per day", "daily limit", ``` ```diff @@ -2697,6 +2700,60 @@ def _is_model_not_found_error(exc: Exception) -> bool: )) +def _is_model_incompatible_error(exc: Exception) -> bool: + """Detect "this route cannot serve this model" 400s (capability mismatch). + + Distinct from :func:`_is_model_not_found_error` (the model does not exist + anywhere): here the model name is valid but the *current provider/account* + is structurally unable to run it. The canonical case is a configured + fallback that cannot run the main model — e.g. an ``openai-codex`` / + ChatGPT-account fallback asked to compress a ``glm-5.2`` conversation:: + + Error code: 400 - {'detail': "The 'glm-5.2' model is not supported + when using Codex with a ChatGPT account."} + + The candidate authenticates fine and builds a client, so the auth and + payment predicates don't fire and the call would otherwise raise and + abort the whole auxiliary task (commonly compression — which then drops + middle turns and churns the session, destroying the prompt cache). + Treating it as a fallback-worthy capability error lets the chain skip the + incapable route and continue to the next candidate, mirroring the + context-window feasibility screen (#52392). + + Billing/quota 400s belong to :func:`_is_payment_error`; "model does not + exist" 400s belong to :func:`_is_model_not_found_error`. This predicate + explicitly excludes both so the three don't overlap. + """ + status = getattr(exc, "status_code", None) + if status not in {400, None}: + return False + err_lower = str(exc).lower() + # Not-found 400s ("invalid model ID", "model does not exist") are owned by + # _is_model_not_found_error. Billing/free-tier 400s are owned by the + # payment path — key on the billing keywords directly here rather than + # calling _is_payment_error(), because that predicate is status-gated + # ({402,403,404,429,None}) and would not recognise a 400-coded billing + # body, letting it leak into this capability bucket. + if _is_model_not_found_error(exc): + return False + if any(kw in err_lower for kw in ( + "credits", "insufficient funds", "billing", "out of funds", + "balance_depleted", "no usable credits", "payment required", + "free tier", "free-tier", "not available on the free tier", + "model_not_supported_on_free_tier", "quota", + )): + return False + return any(kw in err_lower for kw in ( + "is not supported when using", # codex/ChatGPT-account model gating + "model is not supported", + "not supported with this", + "not supported for this account", + "model_not_supported", + "does not support this model", + "unsupported model", + )) + + def _evict_cached_clients(provider: str) -> None: """Drop cached auxiliary clients for a provider so fresh creds are used.""" normalized = _normalize_aux_provider(provider) ``` ```diff @@ -3147,6 +3204,88 @@ def _try_main_agent_model_fallback( return client, resolved_model or main_model, label +# ── Context-window screening for runtime fallback chains (issue #52392) ── +# +# When the runtime auxiliary fallback chain selects a candidate that is +# reachable but has a context window smaller than the compression task +# requires, the call errors out instead of continuing to the next, viable +# candidate. The startup feasibility check in +# ``agent.conversation_compression.check_compression_model_feasibility`` +# already filters too-small auxiliary models at startup, but the runtime +# fallback chain (``_try_configured_fallback_chain`` and +# ``_try_main_fallback_chain``) does not apply the same filter, so +# compression can stop at the first alive door even if the room behind it +# is too small. +# +# The helpers below screen each candidate by its effective context window +# before it is returned. ``None`` results from ``get_model_context_length`` +# are passed through (we cannot prove a model is too small, so we do not +# block it). This preserves the existing fallback surface for +# unrecognised/custom models while closing the gap on the well-known ones. + +def _task_minimum_context_length(task: Optional[str]) -> Optional[int]: + """Return the minimum context length required for an auxiliary task. + + Only ``compression`` carries an explicit minimum today (the same + ``MINIMUM_CONTEXT_LENGTH`` (64K) floor that + ``check_compression_model_feasibility`` already enforces at startup). + Other tasks (``vision``, ``title_generation``, ``web_extract``, + ``skills_hub``, ``mcp``, ``session_search``) return ``None`` — they + have no per-task context floor and the runtime chain must remain + permissive for them. + + Returns ``None`` for an empty/``None`` task name so the helper is a + safe no-op when called from generic sites. + """ + if not task: + return None + if task == "compression": + return MINIMUM_CONTEXT_LENGTH + return None + + +def _candidate_context_window( + provider: str, + model: str, + base_url: str = "", + api_key: str = "", +) -> Optional[int]: + """Resolve the effective context window for a fallback candidate. + + Thin wrapper around :func:`agent.model_metadata.get_model_context_length` + that swallows probe failures (returns ``None``). Callers treat + ``None`` as "unknown — pass through" so the existing fallback + surface is preserved when the context-length resolver chain cannot + determine a value (custom endpoints, models not in the registry, + offline endpoints). + + Best-effort, never raises — the runtime fallback chain must keep + moving even if the resolver hits a probe error. + """ + if not model: + return None + try: + ctx = get_model_context_length( + model, + base_url=base_url, + api_key=[redacted:secret-assignment] + provider=provider, + ) + except Exception as exc: + logger.debug( + "Auxiliary fallback: could not resolve context window for %s/%s: %s", + provider, model, exc, + ) + return None + # ``get_model_context_length`` returns an int (with a 256K default + # fallback when nothing else matches). We still propagate ``None`` if + # a future change returns ``Optional[int]`` — being explicit is + # cheap and the test suite covers both shapes. + if isinstance(ctx, int) and ctx > 0: + return ctx + return None + + def _try_configured_fallback_chain( task: str, failed_provider: str, ``` ```diff @@ -3171,6 +3310,7 @@ def _try_configured_fallback_chain( skip = failed_provider.lower().strip() tried = [] + min_ctx = _task_minimum_context_length(task) for i, entry in enumerate(chain): if not isinstance(entry, dict): ``` ```diff @@ -3188,6 +3328,20 @@ def _try_configured_fallback_chain( fb_client, resolved_model = None, None if fb_client is not None: + if min_ctx is not None and resolved_model: + fb_ctx = _candidate_context_window( + fb_provider, + resolved_model, + base_url=str(entry.get("base_url") or ""), + api_key=[redacted:secret-assignment] or "", + ) + if fb_ctx is not None and fb_ctx < min_ctx: + logger.info( + "Auxiliary %s: skipping %s (%s context=%d < min=%d), continuing chain", + task, label, resolved_model, fb_ctx, min_ctx, + ) + tried.append(f"{label} (context too small: {fb_ctx}<{min_ctx})") + continue logger.info( "Auxiliary %s: %s on %s — configured fallback to %s (%s)", task, reason, failed_provider, label, resolved_model or fb_model or "default", ``` ```diff @@ -3203,6 +3357,28 @@ def _try_configured_fallback_chain( return None, None, "" +def _try_configured_fallback_for_unavailable_client( + task: Optional[str], + failed_provider: str, +) -> Tuple[Optional[Any], Optional[str], str]: + """Try task fallback_chain when an explicit aux provider cannot build. + + This covers the "no client" case before any request is sent: missing + raw env key, unavailable OAuth/pool credentials, or provider resolver + returning ``(None, None)``. It deliberately stops at the configured + per-task fallback chain; the main-agent model remains the last-resort + runtime fallback for request-time capacity errors. + """ + explicit = (failed_provider or "").strip().lower() + if not task or not explicit or explicit in {"auto"}: + return None, None, "" + return _try_configured_fallback_chain( + task, + explicit, + reason="provider unavailable", + ) + + def _fallback_entry_api_key(entry: Dict[str, Any]) -> Optional[str]: """Resolve inline or env-backed API key from a fallback-chain entry.""" explicit = str(entry.get("api_key") or "").strip() ``` ```diff @@ -3261,6 +3437,7 @@ def _try_main_fallback_chain( main_norm = (_read_main_provider() or "").strip().lower() skip = {p for p in (failed_norm, main_norm, "auto") if p} tried: List[str] = [] + min_ctx = _task_minimum_context_length(task) for i, entry in enumerate(chain): if not isinstance(entry, dict): ``` ```diff @@ -3284,6 +3461,20 @@ def _try_main_fallback_chain( logger.debug("Auxiliary %s: main fallback %s failed to resolve: %s", task or "call", label, exc) fb_client, resolved_model = None, None if fb_client is not None: + if min_ctx is not None: + fb_ctx = _candidate_context_window( + fb_provider, + resolved_model or fb_model, + base_url=str(entry.get("base_url") or ""), + api_key=[redacted:secret-assignment] or "", + ) + if fb_ctx is not None and fb_ctx < min_ctx: + logger.info( + "Auxiliary %s: skipping %s (context=%d < min=%d), continuing chain", + task or "call", label, fb_ctx, min_ctx, + ) + tried.append(f"{label} (context too small: {fb_ctx}<{min_ctx})") + continue logger.info( "Auxiliary %s: %s on %s — main fallback chain to %s (%s)", task or "call", reason, failed_provider or "auto", label, ``` ```diff @@ -5344,21 +5535,30 @@ def call_llm( ) if client is None: # When the user explicitly chose a non-OpenRouter provider but no - # credentials were found, fail fast instead of silently routing - # through OpenRouter (which causes confusing 404s). + # credentials were found, honor the task fallback_chain before + # raising. Missing raw env keys are recoverable for auxiliary + # tasks because fallback entries may use OAuth / credential-pool + # auth (for example openai-codex). _explicit = (resolved_provider or "").strip().lower() if _explicit and _explicit not in {"auto", "openrouter", "custom"}: - raise RuntimeError( - f"Provider '{_explicit}' is set in config.yaml but no API key " - f"was found. Set the {_explicit.upper()}_API_KEY environment " - f"variable, or switch to a different provider with `hermes model`." + fb_client, fb_model, fb_label = _try_configured_fallback_for_unavailable_client( + task, _explicit, ) + if fb_client is not None: + client, final_model = fb_client, fb_model + resolved_provider = fb_label or resolved_provider + else: + raise RuntimeError( + f"Provider '{_explicit}' is set in config.yaml but no API key " + f"was found. Set the {_explicit.upper()}_API_KEY environment " + f"variable, or switch to a different provider with `hermes model`." + ) # For auto/custom with no credentials, try the full auto chain # rather than hardcoding OpenRouter (which may be depleted). # Pass model=None so each provider uses its own default — # resolved_model may be an OpenRouter-format slug that doesn't # work on other providers. - if not resolved_base_url: + if client is None and not resolved_base_url: logger.info("Auxiliary %s: provider %s unavailable, trying auto-detection chain", task or "call", resolved_provider) client, final_model = _get_cached_client("auto", main_runtime=main_runtime, task=task) ``` ```diff @@ -5657,6 +5857,7 @@ def call_llm( _is_payment_error(first_err) or _is_connection_error(first_err) or _is_rate_limit_error(first_err) + or _is_model_incompatible_error(first_err) ) # Respect explicit provider choice for transient errors (auth, request # validation, etc.) but allow fallback when the provider clearly cannot ``` ```diff @@ -5667,7 +5868,19 @@ def call_llm( is_auto = resolved_provider in {"auto", "", None} # Capacity errors bypass the explicit-provider gate: the provider # literally cannot serve this request regardless of user intent. - is_capacity_error = _is_payment_error(first_err) or _is_connection_error(first_err) + # Rate limits are included: after retries are exhausted, a 429 means + # the provider cannot serve this request — fall back. See #52228. + # Model-incompatibility 400s are also a hard capability mismatch (the + # route cannot run this model at all — e.g. a codex/ChatGPT-account + # fallback asked to compress a glm-5.2 conversation), so they bypass + # the explicit-provider gate and continue to the next candidate + # instead of aborting the auxiliary task and churning the session. + is_capacity_error = ( + _is_payment_error(first_err) + or _is_connection_error(first_err) + or _is_rate_limit_error(first_err) + or _is_model_incompatible_error(first_err) + ) if should_fallback and (is_auto or is_capacity_error): if _is_payment_error(first_err): reason = "payment error" ``` ```diff @@ -5680,6 +5893,8 @@ def call_llm( ) elif _is_rate_limit_error(first_err): reason = "rate limit" + elif _is_model_incompatible_error(first_err): + reason = "model incompatible with route" else: reason = "connection error" logger.info("Auxiliary %s: %s on %s (%s), trying fallback", ``` ```diff @@ -5854,12 +6069,21 @@ async def async_call_llm( if client is None: _explicit = (resolved_provider or "").strip().lower() if _explicit and _explicit not in {"auto", "openrouter", "custom"}: - raise RuntimeError( - f"Provider '{_explicit}' is set in config.yaml but no API key " - f"was found. Set the {_explicit.upper()}_API_KEY environment " - f"variable, or switch to a different provider with `hermes model`." + fb_client, fb_model, fb_label = _try_configured_fallback_for_unavailable_client( + task, _explicit, ) - if not resolved_base_url: + if fb_client is not None: + client, final_model = _to_async_client( + fb_client, fb_model or "", is_vision=(task == "vision") + ) + resolved_provider = fb_label or resolved_provider + else: + raise RuntimeError( + f"Provider '{_explicit}' is set in config.yaml but no API key " + f"was found. Set the {_explicit.upper()}_API_KEY environment " + f"variable, or switch to a different provider with `hermes model`." + ) + if client is None and not resolved_base_url: logger.info("Auxiliary %s: provider %s unavailable, trying auto-detection chain", task or "call", resolved_provider) client, final_model = _get_cached_client("auto", async_mode=True, main_runtime=main_runtime, task=task) ``` ```diff @@ -6109,12 +6333,22 @@ async def async_call_llm( _is_payment_error(first_err) or _is_connection_error(first_err) or _is_rate_limit_error(first_err) + or _is_model_incompatible_error(first_err) ) - # Capacity errors (payment/quota/connection) bypass the explicit-provider - # gate — the provider cannot serve the request regardless of user intent. + # Capacity errors (payment/quota/connection/rate-limit) bypass the + # explicit-provider gate — the provider cannot serve the request + # regardless of user intent. Rate limits are included: after retries + # are exhausted, a 429 means the provider is at capacity. See #52228. # See #26803: daily token quota must fall back like a 402 credit error. + # Model-incompatibility 400s (route cannot run this model at all) + # bypass the gate too — see the sync call_llm() path for rationale. is_auto = resolved_provider in {"auto", "", None} - is_capacity_error = _is_payment_error(first_err) or _is_connection_error(first_err) + is_capacity_error = ( + _is_payment_error(first_err) + or _is_connection_error(first_err) + or _is_rate_limit_error(first_err) + or _is_model_incompatible_error(first_err) + ) if should_fallback and (is_auto or is_capacity_error): if _is_payment_error(first_err): reason = "payment error" ``` ```diff @@ -6123,6 +6357,8 @@ async def async_call_llm( ) elif _is_rate_limit_error(first_err): reason = "rate limit" + elif _is_model_incompatible_error(first_err): + reason = "model incompatible with route" else: reason = "connection error" logger.info("Auxiliary %s (async): %s on %s (%s), trying fallback", ``` - `agent/conversation_compression.py` ```diff @@ -90,6 +90,7 @@ def check_compression_model_feasibility(agent: Any) -> None: try: from agent.auxiliary_client import ( _resolve_task_provider_model, + _try_configured_fallback_for_unavailable_client, get_text_auxiliary_client, ) from agent.model_metadata import ( ``` ```diff @@ -97,10 +98,6 @@ def check_compression_model_feasibility(agent: Any) -> None: get_model_context_length, ) - client, aux_model = get_text_auxiliary_client( - "compression", - main_runtime=agent._current_main_runtime(), - ) # Best-effort aux provider label for the warning message. The # configured provider may be "auto", in which case we fall back # to the client's base_url hostname so the user can still tell ``` ```diff @@ -109,6 +106,19 @@ def check_compression_model_feasibility(agent: Any) -> None: _aux_cfg_provider, _, _, _, _ = _resolve_task_provider_model("compression") except Exception: _aux_cfg_provider = "" + client, aux_model = get_text_auxiliary_client( + "compression", + main_runtime=agent._current_main_runtime(), + ) + if client is None or not aux_model: + fb_client, fb_model, fb_label = _try_configured_fallback_for_unavailable_client( + "compression", + _aux_cfg_provider, + ) + if fb_client is not None and fb_model: + client, aux_model = fb_client, fb_model + if "(" in fb_label and fb_label.endswith(")"): + _aux_cfg_provider = fb_label.rsplit("(", 1)[1][:-1] if client is None or not aux_model: if _aux_cfg_provider and _aux_cfg_provider != "auto": msg = ( ``` - `scripts/release.py` ```diff @@ -45,6 +45,8 @@ # Auto-extracted from noreply emails + manual overrides AUTHOR_MAP = { + "[redacted:email]": "herbalizer404", # PR #49076 + #51835 salvage (auxiliary compression fallback: 403/session-usage payment errors + honor fallback chain when aux provider auth unavailable) + "[redacted:email]": "pyxl-dev", # PR #52230 salvage (include rate-limit in auxiliary capacity-error fallback gate; #52228) "[redacted:email]": "benbenlijie", # PR #47205 salvage (named custom-provider extra_body + Z.AI Coding overload adaptive backoff; #50663) "[redacted:email]": "Danamove", # PR #46726 salvage (kill venv-resident pythonw gateway before recreating venv on Windows; #47036/#47557/#47910) "[redacted:email]": "wgu9", # PR #51468 salvage (WSL/no-systemd orphan gateway tracking, #51325) ``` - `tests/agent/test_auxiliary_client.py` ```diff @@ -23,6 +23,7 @@ _is_payment_error, _is_rate_limit_error, _is_model_not_found_error, + _is_model_incompatible_error, _refresh_nous_recommended_model, _normalize_aux_provider, _try_payment_fallback, ``` ```diff @@ -1406,6 +1407,21 @@ def test_404_free_tier_model_block_is_payment(self): exc.status_code = 404 assert _is_payment_error(exc) is True + def test_403_subscription_required_is_payment(self): + exc = Exception( + "this model requires a subscription, upgrade for access: " + "https://ollama.com/upgrade" + ) + setattr(exc, "status_code", 403) + assert _is_payment_error(exc) is True + + def test_429_session_usage_limit_is_payment(self): + exc = Exception( + "you have reached your session usage limit, upgrade for higher limits" + ) + setattr(exc, "status_code", 429) + assert _is_payment_error(exc) is True + def test_404_generic_not_found_is_not_payment(self): exc = Exception("Not Found") exc.status_code = 404 ``` ```diff @@ -1527,6 +1543,60 @@ def test_500_is_not_model_not_found(self): assert _is_model_not_found_error(exc) is False +class TestIsModelIncompatibleError: + """_is_model_incompatible_error detects 400s where the route cannot run + the model at all (capability mismatch), distinct from not-found and + payment errors.""" + + def test_codex_chatgpt_account_model_gating(self): + """The exact incident: an openai-codex/ChatGPT-account fallback asked + to compress a glm-5.2 conversation.""" + exc = Exception( + "Error code: 400 - {'detail': \"The 'glm-5.2' model is not " + "supported when using Codex with a ChatGPT account.\"}" + ) + exc.status_code = 400 + assert _is_model_incompatible_error(exc) is True + + def test_model_is_not_supported_phrasing(self): + exc = Exception("This model is not supported for this endpoint") + exc.status_code = 400 + assert _is_model_incompatible_error(exc) is True + + def test_unsupported_model_keyword(self): + exc = Exception("unsupported model for this account tier") + exc.status_code = 400 + assert _is_model_incompatible_error(exc) is True + + def test_not_found_is_not_incompatible(self): + """A model-does-not-exist 400 belongs to _is_model_not_found_error — + the two predicates must not overlap.""" + exc = Exception("openrouter/foo/bar is not a valid model ID") + exc.status_code = 400 + assert _is_model_incompatible_error(exc) is False + assert _is_model_not_found_error(exc) is True + + def test_payment_400_is_not_incompatible(self): + """A billing 400 that also contains capability-ish phrasing must be + rejected here — billing keywords win so the payment path owns it and + the two buckets don't overlap.""" + exc = Exception("insufficient credits: model is not supported on free tier") + exc.status_code = 400 + assert _is_model_incompatible_error(exc) is False + + def test_wrong_status_is_not_incompatible(self): + exc = Exception("model is not supported") # right phrase, wrong status + exc.status_code = 500 + assert _is_model_incompatible_error(exc) is False + + def test_generic_400_is_not_incompatible(self): + """A plain request-validation 400 without capability phrasing must not + trigger fallback (we respect explicit-provider choice for those).""" + exc = Exception("invalid value for parameter temperature") + exc.status_code = 400 + assert _is_model_incompatible_error(exc) is False + + class TestRefreshNousRecommendedModel: """_refresh_nous_recommended_model picks a fresh model after a stale 404.""" ``` ```diff @@ -1870,6 +1940,44 @@ def test_explicit_provider_falls_back_to_main_when_chain_exhausted(self, monkeyp assert main_client.chat.completions.create.called + def test_explicit_provider_rate_limit_triggers_fallback(self, monkeypatch): + """429 rate-limit on an explicit provider must trigger fallback (not be ignored). + + Regression test for #52228: rate limits were excluded from + ``is_capacity_error``, so explicit-provider auxiliary calls never + fell back on 429 — only auto-provider calls did. + """ + monkeypatch.setenv("OPENROUTER_API_KEY", "or-key") + + primary_client = MagicMock() + rate_err = Exception("Rate limit exceeded, try again in 60 seconds") + rate_err.status_code = 429 + primary_client.chat.completions.create.side_effect = rate_err + + fallback_client = MagicMock() + fallback_client.chat.completions.create.return_value = MagicMock(choices=[ + MagicMock(message=MagicMock(content="from fallback chain")) + ]) + + with patch("agent.auxiliary_client._get_cached_client", + return_value=(primary_client, "gpt-5.5")), \ + patch("agent.auxiliary_client._resolve_task_provider_model", + return_value=("openai-codex", "gpt-5.5", None, None, None)), \ + patch("agent.auxiliary_client._try_configured_fallback_chain", + return_value=(fallback_client, "deepseek-v4-pro", "fallback_chain[0](opencode-go)")) as mock_chain, \ + patch("agent.auxiliary_client._try_main_agent_model_fallback") as mock_main: + result = call_llm( + task="kanban_decomposer", + messages=[{"role": "user", "content": "decompose this"}], + ) + + # Fallback chain MUST be tried for rate-limit on explicit provider + mock_chain.assert_called() + assert fallback_client.chat.completions.create.called + # Main agent fallback should NOT be needed when chain succeeds + mock_main.assert_not_called() + + def test_warning_emitted_when_all_fallbacks_exhausted(self, monkeypatch, caplog): """When chain AND main model both fail, a user-visible warning fires before re-raise.""" monkeypatch.setenv("OPENROUTER_API_KEY", "or-key") ``` ```diff @@ -1896,6 +2004,81 @@ def test_warning_emitted_when_all_fallbacks_exhausted(self, monkeypatch, caplog) "all fallbacks exhausted" in r.message for r in caplog.records ), f"Expected exhaustion warning, got: {[r.message for r in caplog.records]}" + def test_explicit_provider_no_client_uses_configured_chain_before_error(self, monkeypatch): + """Missing primary credentials should still honor auxiliary fallback_chain.""" + chain_client = MagicMock() + chain_client.chat.completions.create.return_value = MagicMock(choices=[ + MagicMock(message=MagicMock(content="from configured chain")) + ]) + + with patch("agent.auxiliary_client._get_cached_client", + return_value=(None, None)), \ + patch("agent.auxiliary_client._resolve_task_provider_model", + return_value=("ollama-cloud", "deepseek-v4-flash:cloud", None, None, None)), \ + patch("agent.auxiliary_client._try_configured_fallback_chain", + return_value=(chain_client, "gpt-5.4-mini", "fallback_chain[0](openai-codex)")) as mock_chain: + result = call_llm( + task="compression", + messages=[{"role": "user", "content": "hello"}], + ) + + assert chain_client.chat.completions.create.called + assert result.choices[0].message.content == "from configured chain" + mock_chain.assert_called_once_with( + "compression", + "ollama-cloud", + reason="provider unavailable", + ) + + def test_explicit_provider_no_client_without_chain_keeps_clear_error(self, monkeypatch): + """No fallback configured: keep the existing actionable missing-key error.""" + with patch("agent.auxiliary_client._get_cached_client", + return_value=(None, None)), \ + patch("agent.auxiliary_client._resolve_task_provider_model", + return_value=("ollama-cloud", "deepseek-v4-flash:cloud", None, None, None)), \ + patch("agent.auxiliary_client._try_configured_fallback_chain", + return_value=(None, None, "")) as mock_chain: + with pytest.raises(RuntimeError, match="Provider 'ollama-cloud'.*no API key"): + call_llm( + task="compression", + messages=[{"role": "user", "content": "hello"}], + ) + + mock_chain.assert_called_once_with( + "compression", + "ollama-cloud", + reason="provider unavailable", + ) + + def test_fallback_entry_openai_codex_uses_oauth_pool_without_inline_key(self): + """Configured Codex fallback resolves through Hermes auth / credential pool.""" + from agent.auxiliary_client import _resolve_fallback_entry + + pool_entry = MagicMock() + pool_entry.id = "codex-pool-1" + pool_entry.runtime_api_key = "[redacted:secret-assignment]" + pool_entry.access_token = "[redacted:secret-assignment]" + pool_entry.runtime_base_url = "https://chatgpt.com/backend-api/codex" + + real_client = MagicMock() + real_client.api_key = "[redacted:secret-assignment]" + real_client.base_url = "https://chatgpt.com/backend-api/codex" + + with patch("agent.auxiliary_client._select_pool_entry", + return_value=(True, pool_entry)), \ + patch("agent.auxiliary_client._read_codex_access_token", + side_effect=AssertionError("should use pool token")), \ + patch("agent.auxiliary_client.OpenAI", return_value=real_client) as mock_openai: + client, model = _resolve_fallback_entry({ + "provider": "openai-codex", + "model": "gpt-5.4-mini", + }) + + assert client is not None + assert model == "gpt-5.4-mini" + mock_openai.assert_called_once() + assert mock_openai.call_args.kwargs["api_key"] == "codex-oauth-token" + class TestTryMainAgentModelFallback: """_try_main_agent_model_fallback resolves the user's main provider+model as a safety net.""" ``` ```diff @@ -4070,3 +4253,257 @@ def test_empty_model_falls_back_to_url_only(self): ): assert auxiliary_max_tokens_param(4096, model="") == {"max_tokens": 4096} assert auxiliary_max_tokens_param(4096, model=None) == {"max_tokens": 4096} + + +# ── Regression tests for issue #52392 ───────────────────────────────────── +# Compression fallback chain currently picks the first reachable candidate +# without checking whether the candidate's context window is large enough. +# When the chosen candidate is reachable but too small for the compression +# task, the call errors out instead of continuing through the chain. + +class TestCompressionFallbackContextFilter: + """Aux fallback chains must skip candidates whose context window is + smaller than the task minimum, then continue to the next candidate. + + Layer coverage: + L2: _try_configured_fallback_chain skips too-small candidates + L3: _try_main_fallback_chain skips too-small candidates + L4: candidates with unknown context (None) are passed through + L5: backward compat — first viable candidate still wins + """ + + @staticmethod + def _make_chain_entry(provider, model, base_url="https://example.com/v1", + api_key="k"): + return { + "provider": provider, + "model": model, + "base_url": base_url, + "api_key": api_key, + } + + def _mock_resolve(self, entry): + """Mock _resolve_fallback_entry to return a (client, model) per entry.""" + client = MagicMock() + client.base_url = entry.get("base_url", "") + return client, entry["model"] + + # ── L2: configured fallback chain ───────────────────────────────── + + def test_configured_chain_skips_too_small_candidate_for_compression(self, monkeypatch): + """When entry[0] is reachable but too small and entry[1] is large enough, + _try_configured_fallback_chain must return entry[1], not entry[0].""" + from agent.auxiliary_client import ( + _try_configured_fallback_chain, + ) + + small_client = MagicMock(name="small_client") + large_client = MagicMock(name="large_client") + entries = [ + self._make_chain_entry("small-provider", "tiny-8k"), + self._make_chain_entry("big-provider", "huge-1m"), + ] + + def fake_resolve(entry): + if entry is entries[0]: + return small_client, "tiny-8k" + return large_client, "huge-1m" + + # tiny-8k resolves to 8K (below 64K floor); huge-1m resolves to 1M + def fake_ctx(model, base_url="", api_key="", **kwargs): + return {"tiny-8k": 8192, "huge-1m": 1_048_576}.get(model, 256_000) + + monkeypatch.setattr( + "agent.auxiliary_client._get_auxiliary_task_config", + lambda task: {"fallback_chain": entries} if task == "compression" else {}, + ) + + with patch("agent.auxiliary_client._resolve_fallback_entry", + side_effect=fake_resolve), \ + patch("agent.auxiliary_client.get_model_context_length", + side_effect=fake_ctx): + client, model, label = _try_configured_fallback_chain( + task="compression", failed_provider="auto") + + assert client is large_client, ( + f"Expected large_client (1M context), got {client}. " + "L2 bug: chain returned the first reachable candidate without " + "screening by context window.") + assert model == "huge-1m" + assert "big-provider" in label + + def test_configured_chain_continues_after_skipping_too_small(self, monkeypatch): + """When all small candidates are skipped and only the last is large enough, + the chain still returns it (does not stop after first filter).""" + from agent.auxiliary_client import _try_configured_fallback_chain + + small_client_a = MagicMock(name="small_a") + small_client_b = MagicMock(name="small_b") + large_client = MagicMock(name="large") + entries = [ + self._make_chain_entry("p1", "small-a-32k"), + self._make_chain_entry("p2", "small-b-48k"), + self._make_chain_entry("p3", "large-512k"), + ] + + def fake_resolve(entry): + if entry is entries[0]: + return small_client_a, "small-a-32k" + if entry is entries[1]: + return small_client_b, "small-b-48k" + return large_client, "large-512k" + + def fake_ctx(model, base_url="", api_key="", **kwargs): + return {"small-a-32k": 32_000, + "small-b-48k": 48_000, + "large-512k": 512_000}.get(model, 256_000) + + monkeypatch.setattr( + "agent.auxiliary_client._get_auxiliary_task_config", + lambda task: {"fallback_chain": entries} if task == "compression" else {}, + ) + + with patch("agent.auxiliary_client._resolve_fallback_entry", + side_effect=fake_resolve), \ + patch("agent.auxiliary_client.get_model_context_length", + side_effect=fake_ctx): + client, model, label = _try_configured_fallback_chain( + task="compression", failed_provider="auto") + + assert client is large_client + assert model == "large-512k" + + # ── L3: main fallback chain ──────────────────────────────────────── + + def test_main_chain_skips_too_small_candidate_for_compression(self, monkeypatch): + """Same behaviour for the top-level main-agent fallback chain.""" + from agent.auxiliary_client import ( + _try_main_fallback_chain, + ) + + small_client = MagicMock(name="small_main") + large_client = MagicMock(name="large_main") + + # Mock load_config + get_fallback_chain to return our controlled chain + chain = [ + self._make_chain_entry("p-small", "tiny-16k"), + self._make_chain_entry("p-large", "huge-1m"), + ] + + def fake_resolve(entry): + if entry is chain[0]: + return small_client, "tiny-16k" + return large_client, "huge-1m" + + def fake_ctx(model, base_url="", api_key="", **kwargs): + return {"tiny-16k": 16_384, "huge-1m": 1_048_576}.get(model, 256_000) + + monkeypatch.setattr( + "hermes_cli.fallback_config.get_fallback_chain", + lambda cfg: chain, + ) + + with patch("agent.auxiliary_client._resolve_fallback_entry", + side_effect=fake_resolve), \ + patch("agent.auxiliary_client.get_model_context_length", + side_effect=fake_ctx), \ + patch("agent.auxiliary_client._is_provider_unhealthy", + return_value=False): + client, model, label = _try_main_fallback_chain( + task="compression", failed_provider="auto") + + assert client is large_client, ( + f"Expected large_client (1M), got {client}. " + "L3 bug: main chain returned the first reachable candidate " + "without screening by context window.") + assert model == "huge-1m" + + # ── L4: unknown context passthrough ──────────────────────────────── + + def test_configured_chain_passes_through_unknown_context(self, monkeypatch): + """When get_model_context_length returns None (cannot probe), + the candidate is NOT filtered — the existing behaviour of using + the default 256K fallback in the resolver chain is preserved.""" + from agent.auxiliary_client import _try_configured_fallback_chain + + unknown_client = MagicMock(name="unknown_client") + entries = [self._make_chain_entry("unknown-provider", "unprobed-model")] + + def fake_resolve(entry): + return unknown_client, "unprobed-model" + + def fake_ctx(model, base_url="", api_key="", **kwargs): + return None # cannot determine context length + + monkeypatch.setattr( + "agent.auxiliary_client._get_auxiliary_task_config", + lambda task: {"fallback_chain": entries} if task == "compression" else {}, + ) + + with patch("agent.auxiliary_client._resolve_fallback_entry", + side_effect=fake_resolve), \ + patch("agent.auxiliary_client.get_model_context_length", + side_effect=fake_ctx): + client, model, label = _try_configured_fallback_chain( + task="compression", failed_provider="auto") + + assert client is unknown_client, ( + "L4 bug: candidates with unknown context must be passed through, " + "not blocked. Being unsure is not the same as being too small.") + assert model == "unprobed-model" + + # ── L5: backward compat — non-compression tasks unchanged ────────── + + def test_non_compression_task_does_not_filter_by_context(self, monkeypatch): + """For tasks without a context floor (e.g. title_generation, vision), + the chain behaviour is unchanged: first reachable candidate wins.""" + from agent.auxiliary_client import _try_configured_fallback_chain + + small_client = MagicMock(name="small") + entries = [self._make_chain_entry("p", "tiny-4k")] + + def fake_resolve(entry): + return small_client, "tiny-4k" + + def fake_ctx(model, base_url="", api_key="", **kwargs): + return 4_096 # small — but title_generation has no floor + + monkeypatch.setattr( + "agent.auxiliary_client._get_auxiliary_task_config", + lambda task: {"fallback_chain": entries} if task == "title_generation" else {}, + ) + + with patch("agent.auxiliary_client._resolve_fallback_entry", + side_effect=fake_resolve), \ + patch("agent.auxiliary_client.get_model_context_length", + side_effect=fake_ctx): + client, model, label = _try_configured_fallback_chain( + task="title_generation", failed_provider="auto") + + assert client is small_client, ( + "L5 regression: non-compression tasks must not be filtered " + "by context window. The first reachable candidate should win.") + assert model == "tiny-4k" + + # ── End-to-end: configured chain skips too-small for vision too ── + # vision has its own implicit context requirements; test that the + # compression-specific filter does NOT affect vision chains. + + def test_compression_task_uses_minimum_context_constant(self): + """The task minimum for compression must equal MINIMUM_CONTEXT_LENGTH + so the runtime fallback stays consistent with the startup feasibility + check in agent/conversation_compression.py.""" + from agent.auxiliary_client import _task_minimum_context_length + from agent.model_metadata import MINIMUM_CONTEXT_LENGTH + + assert _task_minimum_context_length("compression") == MINIMUM_CONTEXT_LENGTH + # Non-compression tasks have no minimum (None) + assert _task_minimum_context_length("vision") is None + assert _task_minimum_context_length("title_generation") is None + assert _task_minimum_context_length("web_extract") is None + assert _task_minimum_context_length("skills_hub") is None + assert _task_minimum_context_length("mcp") is None + assert _task_minimum_context_length("session_search") is None + # Empty / unknown tasks have no minimum + assert _task_minimum_context_length("") is None + assert _task_minimum_context_length(None) is None ``` - `tests/run_agent/test_compression_feasibility.py` ```diff @@ -306,6 +306,39 @@ def test_warns_when_no_auxiliary_provider(mock_get_client): assert agent._compression_warning is not None +def test_no_unavailable_warning_when_configured_fallback_chain_resolves(): + """Primary compression provider can be down if configured fallback works.""" + agent = _make_agent(main_context=200_000, threshold_percent=0.50) + fallback_client = MagicMock() + fallback_client.base_url = "https://chatgpt.com/backend-api/codex" + fallback_client.api_key = "[redacted:secret-assignment]" + + messages = [] + agent._emit_status = lambda msg: messages.append(msg) + + with patch( + "agent.auxiliary_client._resolve_task_provider_model", + return_value=("ollama-cloud", "deepseek-v4-flash:cloud", None, None, None), + ), patch( + "agent.auxiliary_client.get_text_auxiliary_client", + return_value=(None, None), + ), patch( + "agent.auxiliary_client._try_configured_fallback_for_unavailable_client", + return_value=(fallback_client, "gpt-5.4-mini", "fallback_chain[0](openai-codex)"), + ) as mock_fallback, patch( + "agent.model_metadata.get_model_context_length", + return_value=200_000, + ) as mock_ctx_len: + agent._check_compression_model_feasibility() + + assert messages == [] + assert agent._compression_warning is None + mock_fallback.assert_called_once_with("compression", "ollama-cloud") + mock_ctx_len.assert_called_once() + assert mock_ctx_len.call_args.args == ("gpt-5.4-mini",) + assert mock_ctx_len.call_args.kwargs["provider"] == "openai-codex" + + def test_skips_check_when_compression_disabled(): """No check performed when compression is disabled.""" agent = _make_agent(compression_enabled=False) ``` - **#93455: fix(classifier): 429 quota walls route to billing across providers; reset signals stay retryable (consolidates #39441, #63021, #74785)**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `agent/error_classifier.py`, `tests/agent/test_error_classifier.py` - **#25843: fix(auxiliary): trigger provider fallback on 5xx server errors (#25822)**. open - maintainer direction: category `uncategorized`; implication `needs_llm` - files: `agent/auxiliary_client.py` - **#26152: fix(auxiliary): fall back to alternative provider on 5xx server errors**. open - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#25843`, `#25822`, `#26152`, `#26002` - files: `agent/auxiliary_client.py`, `tests/agent/test_auxiliary_client.py` - **#26809: fix(auxiliary): detect quota keywords in _is_payment_error and allow fallback for explicit providers**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#27625`, `#26811` - **#26811: fix(auxiliary): detect quota exhaustion as payment error; allow capacity-error fallback for explicit providers**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `24c209f11`, `#27625`, `#26998` - **#36280: fix(agent): trigger fallback on quota-exhaustion 429s (e.g. Kimi monthly limit)**. open - maintainer direction: category `uncategorized`; implication `needs_llm` - files: `agent/agent_runtime_helpers.py`, `agent/error_classifier.py`, `tests/agent/test_error_classifier.py` - **#39441: fix(agent): classify quota-exhaustion 429s as billing**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `0c3a50753`, `#93455` - **#51002: fix(auxiliary): accept legacy fallback_provider/fallback_model in auxiliary task fallback chain**. open - maintainer direction: category `uncategorized`; implication `needs_llm` - files: `agent/auxiliary_client.py`, `tests/agent/test_auxiliary_client.py` - **#52230: fix: include rate-limit in auxiliary capacity-error fallback gate**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `751adfa6b`, `#52661` - **#52233: fix(agent): fall back on explicit auxiliary rate limits**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#52230`, `#52230`, `#52228`, `#52251`, `#52230`, `#52230`, `#52230`, `#52230` - **#52251: fix(agent): include rate-limit errors in auxiliary capacity-error gate**. closed without merge - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#52230`, `#52230`, `#52233`, `#52228`, `#52230`, `#52233`, `#52228` - **#57115: fix(vision): make fallback chain iterate all entries instead of stopping at first**. open - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `d42e9b178` - files: `agent/auxiliary_client.py`, `tests/agent/test_auxiliary_main_first.py`, `tests/agent/test_auxiliary_vision_fallback_chain.py`, `tests/agent/test_vision_routing_31179.py` - **#65633: fix: classify 429 quota-exhaustion errors as non-retryable for fallback**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `0c3a50753`, `#93455`, `#39441` - **#97572: fix(auxiliary): skip same-provider retry on a vision full-budget timeout**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `c4e394cdf8`, `f50b5bb0fa`, `#101570` ## Fix status - Merged to main in PR #27625 (cited above). No release data is tracked. Verify downstream. - Fix last verified (merged) `2026-08-24`. - Subsystem: `agent`