NousResearch/hermes-agent
max_tokens cap errors causing infinite compression loops
Cluster 55546. Data as of
Nothing open on GitHub. Fix merged in #55570. Check it covers your case.
The short answer
0open issues
0open PRs
5items grouped
- Known?
- Yes. 5 related items grouped here, 1 issue already closed.
- Fix event
- A fix was merged to main in PR #55570. Check it covers your case. No release data is tracked.
- Tracker
- The maintainers' tracker marks this
resolved, deciding PR #64009. That is separate from the fix event. - Still open
- Nothing open on GitHub.
These issues and fixes address cases where provider errors about token limits (400s from DashScope/Qwen and similar) or affordability (402) are misclassified, triggering repeated context compression death-loops or dropped requests instead of recoverable retries.
Open issues0
Open PRs0
Fixes on record
-
#55570 fix(agent): stop over-cap max_tokens 400s from death-looping into compression
files:
agent/conversation_loop.py,agent/model_metadata.py,tests/test_output_cap_parsing.pyShow the diff
-
agent/conversation_loop.py@@ -52,6 +52,7 @@ estimate_messages_tokens_rough, estimate_request_tokens_rough, get_context_length_from_provider_error, + is_output_cap_error, parse_available_output_tokens_from_error, save_context_length, ) @@ -3213,6 +3214,45 @@ def _perform_api_call(next_api_kwargs): _retry.restart_with_compressed_messages = True break + # The error is output-cap-shaped (about max_tokens being + # too large) but the provider's wording didn't let us parse + # the available output budget. Compression CANNOT help here + # — the input already fits; the call fails deterministically + # on the oversized max_tokens. Routing it into compression + # re-sends the same max_tokens, gets the identical 400, and + # death-loops until "cannot compress further" (#55546). + # Fail fast with an actionable message instead of looping. + if is_output_cap_error(error_msg): + agent._flush_status_buffer() + agent._vprint( + f"{agent.log_prefix}❌ The provider rejected the request because " + f"max_tokens exceeds its output cap for this model.", + force=True, + ) + agent._vprint( + f"{agent.log_prefix} 💡 Lower model.max_tokens in your config.yaml to " + f"at or below the model's max-output limit. " + f"(This is an output-cap error, not a context overflow — " + f"compression cannot fix it.)", + force=True, + ) + logger.error( + f"{agent.log_prefix}Output-cap error not routed into compression " + f"(max_tokens over provider cap): {error_msg[:200]}" + ) + agent._persist_session(messages, conversation_history) + return { + "messages": messages, + "completed": False, + "api_calls": api_call_count, + "error": ( + "max_tokens exceeds the provider's output cap for this model. " + "Lower model.max_tokens in config.yaml." + ), + "partial": True, + "failed": True, + } + # Error is about the INPUT being too large. Only reduce # context_length when the provider explicitly reports the # real lower limit. If the provider only says "input -
agent/model_metadata.py@@ -1075,10 +1075,29 @@ def parse_available_output_tokens_from_error(error_msg: str) -> Optional[int]: "maximum context length" in error_lower and "requested" in error_lower and "output tokens" in error_lower + ) or ( + # DashScope / Alibaba Cloud (Qwen) phrasing. The provider rejects an + # over-cap output request with a bounded range whose upper bound IS the + # real max-output cap, e.g. + # "Range of max_tokens should be [1, 65536]" + # The input itself fits — this is purely an output-cap error, so reduce + # max_tokens and retry; do NOT compress. + "range of max_tokens should be" in error_lower ) if not is_output_cap_error: return None + # DashScope / Alibaba range form: "Range of max_tokens should be [1, 65536]". + # The upper bound is the available output cap. + _m_range = re.search( + r'range of max_tokens should be\s*\[\s*\d+\s*,\s*(\d+)\s*\]', + error_lower, + ) + if _m_range: + _cap = int(_m_range.group(1)) + if _cap >= 1: + return _cap + # Extract the available_tokens figure. # Anthropic format: "… = available_tokens: 10000" patterns = [ @@ -1125,6 +1144,70 @@ def parse_available_output_tokens_from_error(error_msg: str) -> Optional[int]: return None +def is_output_cap_error(error_msg: str) -> bool: + """Return True if a 400 is about the OUTPUT cap (max_tokens) being too large. + + This is the broader sibling of :func:`parse_available_output_tokens_from_error`: + that function only returns a number when it can extract the available output + budget from a *known* provider phrasing. This one answers the cheaper + yes/no question — "is this an output-cap error at all?" — across providers + whose exact wording we may not yet parse a number from. + + Why this matters: an output-cap 400 is deterministic (every retry with the + same ``max_tokens`` gets the identical rejection). If such an error is + misclassified as a context-overflow it gets routed into the compression + loop, the compressor re-issues the call with the same oversized + ``max_tokens``, the provider rejects it identically, and the session + death-loops until "cannot compress further" (issue #55546, DashScope/Qwen: + "Range of max_tokens should be [1, 65536]"). Compression cannot help an + output-cap error — the input already fits. + + The signal: the error talks about ``max_tokens`` (or its aliases) as a + cap/range/limit, and does NOT talk about the INPUT/prompt/context window + being too long. When both are present we defer to the context-overflow + path (a real input overflow can also mention max_tokens). + """ + error_lower = error_msg.lower() + + mentions_output_param = ( + "max_tokens" in error_lower + or "max_output_tokens" in error_lower + or "max_completion_tokens" in error_lower + ) + if not mentions_output_param: + return False + + # Phrasing that signals the OUTPUT cap specifically is the problem. + output_cap_signal = ( + "range of max_tokens should be" in error_lower # DashScope / Alibaba + or "available_tokens" in error_lower # Anthropic + or "available tokens" in error_lower + or ("in the output" in error_lower # OpenRouter / Nous + and "maximum context length" in error_lower) + or ("requested" in error_lower # LM Studio / llama.cpp + and "output tokens" in error_lower) + or "should be" in error_lower # generic "max_tokens should be <= N" + or "less than or equal" in error_lower + or "must be" in error_lower + ) + if not output_cap_signal: + return False + + # If the error ALSO clearly describes an oversized INPUT, it is a genuine + # context overflow that happens to mention max_tokens — let the + # context-overflow path handle it (it can compress the input). + input_overflow_signal = ( + "prompt is too long" in error_lower + or "prompt too long" in error_lower + or "input is too long" in error_lower + or "input token" in error_lower + or "prompt length" in error_lower + or "prompt contains" in error_lower + or "reduce the length" in error_lower + ) + return not input_overflow_signal + + def _model_id_matches(candidate_id: str, lookup_model: str) -> bool: """Return True if *candidate_id* (from server) matches *lookup_model* (configured). -
tests/test_output_cap_parsing.py@@ -1,5 +1,8 @@ import pytest -from agent.model_metadata import parse_available_output_tokens_from_error +from agent.model_metadata import ( + is_output_cap_error, + parse_available_output_tokens_from_error, +) class TestParseOpenRouterOutputCap: @@ -62,3 +65,58 @@ def test_char_based_no_room_returns_none(self): msg = ("maximum context length is 1000 tokens. However, you requested " "1000 output tokens and your prompt contains 9000 characters.") assert parse_available_output_tokens_from_error(msg) is None + + +class TestParseDashScopeOutputCap: + """DashScope / Alibaba Cloud (Qwen) reject an over-cap output request with + a bounded range whose upper bound is the real max-output cap (#55546).""" + + def test_dashscope_range_format(self): + msg = ("HTTP 400: InternalError.Algo.InvalidParameter: " + "Range of max_tokens should be [1, 65536]") + assert parse_available_output_tokens_from_error(msg) == 65536 + + def test_dashscope_range_arbitrary_bound(self): + msg = "Range of max_tokens should be [1, 8192]" + assert parse_available_output_tokens_from_error(msg) == 8192 + + def test_dashscope_range_with_spaces(self): + msg = "range of max_tokens should be [ 1 , 32768 ]" + assert parse_available_output_tokens_from_error(msg) == 32768 + + +class TestIsOutputCapError: + """`is_output_cap_error` is the broader yes/no gate that keeps an + output-cap 400 out of the compression death-loop even when we can't parse + a number from the provider's wording (#55546).""" + + def test_dashscope_is_output_cap(self): + assert is_output_cap_error( + "Range of max_tokens should be [1, 65536]" + ) is True + + def test_unknown_numeric_max_tokens_cap_is_output_cap(self): + # Provider we don't yet parse a number from, but clearly an output cap. + assert is_output_cap_error("Invalid value: max_tokens should be <= 8192") is True + + def test_anthropic_available_tokens_is_output_cap(self): + assert is_output_cap_error( + "max_tokens: 32768 > context_window: 200000 - " + "input_tokens: [redacted:secret-assignment] = available_tokens: 10000" + ) is True + + def test_real_input_overflow_is_not_output_cap(self): + # Mentions max_tokens but the INPUT is the problem -> compression path. + assert is_output_cap_error( + "prompt is too long: 250000 tokens > 200000 max_tokens window" + ) is False + + def test_gpt5_unsupported_param_is_not_output_cap(self): + # format_error caught earlier; must NOT be treated as an output cap. + assert is_output_cap_error( + "Unsupported parameter: 'max_tokens' is not supported with this " + "model. Use 'max_completion_tokens' instead." + ) is False + + def test_unrelated_error_is_not_output_cap(self): + assert is_output_cap_error("some unrelated 400 error") is False
-
-
#64009 fix: make output-cap retries respect current request capacity
files:
agent/conversation_loop.py,scripts/release.py,tests/run_agent/test_run_agent.pyShow the diff
-
agent/conversation_loop.py@@ -3061,14 +3061,24 @@ def _perform_api_call(next_api_kwargs): # (``/new``), switch to a larger-context model, or reduce # attachments. Forced compaction via ``/compress`` # (``force=True``) is unaffected — it never reaches this loop. + # + # Output-cap errors (max_tokens too large) are NOT input + # overflow — the recovery is a max_tokens-only retry that + # does not require compression. Exempt them from this guard + # so the retry still fires even when compression is disabled. _overflow_reasons = { FailoverReason.long_context_tier, FailoverReason.payload_too_large, FailoverReason.context_overflow, } + _is_output_cap_error = ( + is_output_cap_error(error_msg) + or parse_available_output_tokens_from_error(error_msg) is not None + ) if ( classified.reason in _overflow_reasons and not getattr(agent, "compression_enabled", True) + and not _is_output_cap_error ): agent._flush_status_buffer() agent._vprint( @@ -3472,15 +3482,33 @@ def _perform_api_call(next_api_kwargs): # context_length = total window (input + output combined). available_out = parse_available_output_tokens_from_error(error_msg) if available_out is not None: - # Error is purely about the output cap being too large. - # Cap output to the available space and retry without - # touching context_length or triggering compression. - safe_out = max(1, available_out - 64) # small safety margin + # This is an output-cap error, not input overflow. + # The provider's available_tokens is the authoritative + # cap for the failed request, so keep it as an upper + # bound. Also estimate the current API request shape + # (system prompt, injected context, tool schemas) because + # Hermes may add API-only content not present in persisted + # messages. Use the smaller budget and apply a small + # safety margin. Do not alter context_length. + request_input_estimate = estimate_request_tokens_rough( + api_messages, tools=agent.tools or None, + ) + local_available_out = old_ctx - request_input_estimate + if local_available_out > 0: + safe_out = max(1, min(available_out, local_available_out) - 64) + else: + # The rough local estimate can overshoot the real + # request size. Fall back to the provider-reported + # budget, which is authoritative for the failed + # request. + safe_out = max(1, available_out - 64) agent._ephemeral_max_output_tokens = [redacted:secret-assignment] agent._buffer_vprint( f"⚠️ Output cap too large for current prompt — " f"retrying with max_tokens=[redacted:secret-assignment] " - f"(available_tokens=[redacted:secret-assignment] context_length unchanged at {old_ctx:,})" + f"(provider_available={available_out:,}, " + f"estimated_request_tokens=[redacted:secret-assignment] " + f"context_length unchanged at {old_ctx:,})" ) # Still count against compression_attempts so we don't # loop forever if the error keeps recurring. -
scripts/release.py@@ -136,6 +136,7 @@ "[redacted:email]": "testingbuddies24", # PR #43192 salvage (strip orphan think-tag close tags in progressive gateway stream so a bare </think> whose open was dropped upstream can't leak to the user) "[redacted:email]": "Lazymonter", # PR #42914 salvage (retry launchd bootstrap after bootout on EIO for install/start instead of degrading to detached) "[redacted:email]": "WXBR", # PR #46183 salvage (persist recovered final_response at the finalize_turn chokepoint so recovery-path breaks don't drop the delivered assistant row) + "[redacted:email]": "dmabry", # PR #63862 salvage (output-cap retry: use provider available_tokens + request estimate; exempt parseable vLLM/LM Studio errors from compression-disabled guard) "[redacted:email]": "SahilRakhaiya05", # PR #44073 salvage (fail-closed gateway/external-surface hardening: own-policy defaults, open-policy startup guard, profile-aware multiplex authz, API-server auth, execute_code per-session RPC token) "[redacted:email]": "itenev", # PR #22753 salvage (asyncify model-context resolution in gateway message path so blocking requests.get can't starve Discord heartbeats) "[redacted:email]": "arthurzhang", # PR #34718 salvage (redact Slack App-Level xapp- tokens in agent/redact.py + gateway/run.py) -
tests/run_agent/test_run_agent.py@@ -5355,6 +5355,281 @@ def test_no_kanban_block_when_not_in_kanban_mode(self, agent, monkeypatch): "_record_task_failure should not be called outside kanban mode" ) + # ── Output-cap retry: safe_out uses provider available_out + request estimate ── + + def test_output_cap_retry_uses_provider_available_out(self, agent): + """run_conversation retries an output-cap error with max_tokens <= + available_out - 64, and does NOT halve context_length or trigger + compression. + """ + self._setup_agent(agent) + agent.api_mode = "chat_completions" + agent.provider = "openrouter" + agent.model = "some/model" + agent.max_tokens = [redacted:secret-assignment] + agent.compression_enabled = True + agent.context_compressor.context_length = 200_000 + agent.context_compressor.should_compress = MagicMock(return_value=False) + + error_msg = ( + "max_tokens: 65536 > context_window: 200000 " + "- input_tokens: [redacted:secret-assignment] = available_tokens: 1000" + ) + exc = Exception(error_msg) + exc.status_code = 400 + exc.code = 400 + + ok_resp = _mock_response(content="done", finish_reason="stop") + agent.client.chat.completions.create.side_effect = [exc, ok_resp] + + mock_compress = MagicMock() + with ( + patch.object(agent, "_persist_session"), + patch.object(agent, "_save_trajectory"), + patch.object(agent, "_cleanup_task_resources"), + patch.object(agent.context_compressor, "update_model"), + patch.object(agent, "_compress_context", mock_compress), + ): + result = agent.run_conversation("hello") + + second_call = agent.client.chat.completions.create.call_args_list[1].kwargs + assert result["completed"] is True + assert second_call["max_tokens"] <= 936 + assert agent.context_compressor.context_length == 200_000 + mock_compress.assert_not_called() + + def test_output_cap_retry_with_large_api_only_content(self, agent): + """When a large system prompt makes api_messages huge while persisted + messages stay tiny, the retry cap must still respect provider + available_tokens — not blow up to the full context window. + """ + self._setup_agent(agent) + agent.api_mode = "chat_completions" + agent.provider = "openrouter" + agent.model = "some/model" + agent.max_tokens = [redacted:secret-assignment] + agent.compression_enabled = True + agent.context_compressor.context_length = 200_000 + agent.context_compressor.should_compress = MagicMock(return_value=False) + + # Huge API-only system prompt; persisted messages are tiny. + agent._cached_system_prompt = "S" * 796_000 + + error_msg = ( + "max_tokens: 65536 > context_window: 200000 " + "- input_tokens: [redacted:secret-assignment] = available_tokens: 1000" + ) + exc = Exception(error_msg) + exc.status_code = 400 + exc.code = 400 + + ok_resp = _mock_response(content="done", finish_reason="stop") + agent.client.chat.completions.create.side_effect = [exc, ok_resp] + + mock_compress = MagicMock() + with ( + patch.object(agent, "_persist_session"), + patch.object(agent, "_save_trajectory"), + patch.object(agent, "_cleanup_task_resources"), + patch.object(agent.context_compressor, "update_model"), + patch.object(agent, "_compress_context", mock_compress), + ): + result = agent.run_conversation("hello") + + second_call = agent.client.chat.completions.create.call_args_list[1].kwargs + assert result["completed"] is True + # The current branch (messages-only estimate) would send max_tokens + # near 199927 — this test fails on it. + assert second_call["max_tokens"] <= 936 + assert agent.context_compressor.context_length == 200_000 + mock_compress.assert_not_called() + + def test_output_cap_retry_request_pressure_lower_bound(self, agent): + """When the provider reports a large available_tokens but local request + pressure leaves less room, the retry cap is the smaller of the two. + """ + self._setup_agent(agent) + agent.api_mode = "chat_completions" + agent.provider = "openrouter" + agent.model = "some/model" + agent.max_tokens = [redacted:secret-assignment] + agent.compression_enabled = True + agent.context_compressor.context_length = 200_000 + agent.context_compressor.should_compress = MagicMock(return_value=False) + + # A large API-only system prompt so the local estimate is the binding + # constraint, not the provider's available_tokens. + agent._cached_system_prompt = "S" * 796_000 + + error_msg = ( + "max_tokens: 65536 > context_window: 200000 " + "- input_tokens: [redacted:secret-assignment] = available_tokens: 50000" + ) + exc = Exception(error_msg) + exc.status_code = 400 + exc.code = 400 + + ok_resp = _mock_response(content="done", finish_reason="stop") + agent.client.chat.completions.create.side_effect = [exc, ok_resp] + + mock_compress = MagicMock() + with ( + patch.object(agent, "_persist_session"), + patch.object(agent, "_save_trajectory"), + patch.object(agent, "_cleanup_task_resources"), + patch.object(agent.context_compressor, "update_model"), + patch.object(agent, "_compress_context", mock_compress), + ): + result = agent.run_conversation("hello") + + first_call = agent.client.chat.completions.create.call_args_list[0].kwargs + second_call = agent.client.chat.completions.create.call_args_list[1].kwargs + assert result["completed"] is True + + # Verify the local estimate is actually the lower bound. + from agent.model_metadata import estimate_request_tokens_rough + estimated_request = estimate_request_tokens_rough( + first_call["messages"], tools=agent.tools or None, + ) + local_available = 200_000 - estimated_request + expected_cap = max(1, min(50_000, local_available) - 64) + assert local_available < 50_000 + assert second_call["max_tokens"] == expected_cap + assert agent.context_compressor.context_length == 200_000 + mock_compress.assert_not_called() + + def test_output_cap_retry_safety_floor_at_one(self, agent): + """When provider available_tokens is 1, the retry cap is floored at 1.""" + self._setup_agent(agent) + agent.api_mode = "chat_completions" + agent.provider = "openrouter" + agent.model = "some/model" + agent.max_tokens = [redacted:secret-assignment] + agent.compression_enabled = True + agent.context_compressor.context_length = 200_000 + agent.context_compressor.should_compress = MagicMock(return_value=False) + + error_msg = ( + "max_tokens: 65536 > context_window: 200000 " + "- input_tokens: [redacted:secret-assignment] = available_tokens: 1" + ) + exc = Exception(error_msg) + exc.status_code = 400 + exc.code = 400 + + ok_resp = _mock_response(content="done", finish_reason="stop") + agent.client.chat.completions.create.side_effect = [exc, ok_resp] + + mock_compress = MagicMock() + with ( + patch.object(agent, "_persist_session"), + patch.object(agent, "_save_trajectory"), + patch.object(agent, "_cleanup_task_resources"), + patch.object(agent.context_compressor, "update_model"), + patch.object(agent, "_compress_context", mock_compress), + ): + result = agent.run_conversation("hello") + + second_call = agent.client.chat.completions.create.call_args_list[1].kwargs + assert result["completed"] is True + assert second_call["max_tokens"] == 1 + assert agent.context_compressor.context_length == 200_000 + mock_compress.assert_not_called() + + def test_output_cap_retry_with_compression_disabled(self, agent): + """Output-cap retry must still work when compression.enabled is false. + The recovery is a max_tokens-only retry — it does not require compression, + so the compression-disabled guard must not block it. + """ + self._setup_agent(agent) + agent.api_mode = "chat_completions" + agent.provider = "openrouter" + agent.model = "some/model" + agent.max_tokens = [redacted:secret-assignment] + agent.compression_enabled = False + agent.context_compressor.context_length = 200_000 + agent.context_compressor.should_compress = MagicMock(return_value=False) + + error_msg = ( + "max_tokens: 65536 > context_window: 200000 " + "- input_tokens: [redacted:secret-assignment] = available_tokens: 1000" + ) + exc = Exception(error_msg) + exc.status_code = 400 + exc.code = 400 + + ok_resp = _mock_response(content="done", finish_reason="stop") + agent.client.chat.completions.create.side_effect = [exc, ok_resp] + + mock_compress = MagicMock() + with ( + patch.object(agent, "_persist_session"), + patch.object(agent, "_save_trajectory"), + patch.object(agent, "_cleanup_task_resources"), + patch.object(agent.context_compressor, "update_model"), + patch.object(agent, "_compress_context", mock_compress), + ): + result = agent.run_conversation("hello") + + # Two API calls: the failed one and the retried one. + assert len(agent.client.chat.completions.create.call_args_list) == 2 + second_call = agent.client.chat.completions.create.call_args_list[1].kwargs + assert result["completed"] is True + assert result.get("compaction_disabled") is None + assert second_call["max_tokens"] <= 936 + assert agent.context_compressor.context_length == 200_000 + mock_compress.assert_not_called() + + def test_output_cap_retry_with_compression_disabled_vllm_format(self, agent): + """vLLM/LM Studio error messages contain 'prompt contains ... input + tokens' which is_output_cap_error() treats as an input-overflow signal + (returns False). But parse_available_output_tokens_from_error() CAN + extract a valid available_tokens from them. The compression-disabled + guard must exempt these too — otherwise users on vLLM/LM Studio with + compression off get a terminal failure instead of a max-tokens retry. + """ + self._setup_agent(agent) + agent.api_mode = "chat_completions" + agent.provider = "openrouter" + agent.model = "some/model" + agent.max_tokens = [redacted:secret-assignment] + agent.compression_enabled = False + agent.context_compressor.context_length = 131_072 + agent.context_compressor.should_compress = MagicMock(return_value=False) + + # vLLM-format error (from tests/test_output_cap_parsing.py) + error_msg = ( + "This model's maximum context length is 131072 tokens. " + "However, you requested 1024 output tokens and your prompt " + "contains at least 65537 input tokens, for a total of at least " + "66561 tokens." + ) + exc = Exception(error_msg) + exc.status_code = 400 + exc.code = 400 + + ok_resp = _mock_response(content="done", finish_reason="stop") + agent.client.chat.completions.create.side_effect = [exc, ok_resp] + + mock_compress = MagicMock() + with ( + patch.object(agent, "_persist_session"), + patch.object(agent, "_save_trajectory"), + patch.object(agent, "_cleanup_task_resources"), + patch.object(agent.context_compressor, "update_model"), + patch.object(agent, "_compress_context", mock_compress), + ): + result = agent.run_conversation("hello") + + assert len(agent.client.chat.completions.create.call_args_list) == 2 + second_call = agent.client.chat.completions.create.call_args_list[1].kwargs + assert result["completed"] is True + assert result.get("compaction_disabled") is None + # parse_available_output_tokens_from_error returns 65535 for this message + assert second_call["max_tokens"] <= 65471 # 65535 - 64 + assert agent.context_compressor.context_length == 131_072 + mock_compress.assert_not_called() + class TestHookPayloadSanitizesSimpleNamespace: """Regression: ``_hook_jsonable`` referenced ``SimpleNamespace`` without
-
2 PRs closed without merging
From the close comments: merged via another change 1 · superseded by another PR 1
Closed issues · 1
- Closed#55546[Bug]: max_tokens over provider cap death-loops into context compression when the provider's 400 wording isn't recognized (e.g. DashScope/Qwen)
How this cluster was grouped
-
Link evidence (
structural links throughout): every member is connected by structural evidence: shared changed files, or a closing/superseded reference. -
Pairwise link density
0.40: the share of item pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36.1.00means every pair is linked.
Pick up a Lab task
Check whether a fix covers the open issues here, with your own agent. Bring back evidence for maintainers.
Open the Lab