Reasoning model stale-stream timeout floors and fixes

Fixes on record

Merged to main. A stamp counts the issues a fix is linked to.

• #52795 fix(agent): route reasoning-model thinking-timeouts to timeout, not phantom context_overflow (#52271)

  merged to main

  files: agent/conversation_loop.py, agent/error_classifier.py, agent/reasoning_timeouts.py, agent/thinking_timeout_guidance.py, scripts/release.py, tests/agent/test_thinking_timeout_guidance.py

  Show the diff

  • agent/conversation_loop.py

    @@ -3527,6 +3527,65 @@ def _perform_api_call(next_api_kwargs):
                                 force=True,
                             )
     
    +                    # Detect thinking-timeout pattern: a known reasoning model
    +                    # hit a transport-layer error before the first content
    +                    # token arrived.  Distinct from _is_stream_drop above
    +                    # (which fires for large file-write stream drops) and
    +                    # from any classifier reason that's not a transport
    +                    # timeout.  Reuses the reasoning-model allowlist from
    +                    # agent/reasoning_timeouts.py (Fixes #52217) so the
    +                    # trigger is consistent with what the per-model
    +                    # stale-timeout floor covers.  After the classifier
    +                    # override at agent/error_classifier.py:720-738 (this
    +                    # PR), transport disconnects on reasoning models route
    +                    # to FailoverReason.timeout rather than
    +                    # context_overflow, so this branch actually fires.
    +                    # Detection and message text live in
    +                    # agent.thinking_timeout_guidance so they're
    +                    # unit-testable without driving the full retry loop.
    +                    # (Part 2 of Fixes #52310.)
    +                    from agent.thinking_timeout_guidance import (
    +                        is_thinking_timeout,
    +                    )
    +                    _is_thinking_timeout = is_thinking_timeout(
    +                        classified,
    +                        _model,
    +                        error_msg,
    +                    )
    +                    if _is_thinking_timeout:
    +                        agent._vprint(
    +                            f"{agent.log_prefix}   💡 The model's thinking "
    +                            f"phase exceeded the upstream proxy's idle "
    +                            f"timeout before the first content token "
    +                            f"arrived. This is a known issue with "
    +                            f"reasoning models behind cloud gateways "
    +                            f"(NVIDIA NIM, OpenAI, Anthropic, DeepSeek).",
    +                            force=True,
    +                        )
    +                        agent._vprint(
    +                            f"{agent.log_prefix}      Workarounds in priority order:",
    +                            force=True,
    +                        )
    +                        agent._vprint(
    +                            f"{agent.log_prefix}      1. Set "
    +                            f"`providers.{_provider}.models.{_model}.stale_timeout_seconds: 900` "
    +                            f"in `~/.hermes/config.yaml` to extend the per-call "
    +                            f"timeout. (Hermes's built-in floor is 600s for "
    +                            f"known reasoning models — if you still see this "
    +                            f"after raising, the upstream cap is even shorter.)",
    +                            force=True,
    +                        )
    +                        agent._vprint(
    +                            f"{agent.log_prefix}      2. Lower `reasoning_budget` or set "
    +                            f"`reasoning_effort: medium` on this model if the provider supports it.",
    +                            force=True,
    +                        )
    +                        agent._vprint(
    +                            f"{agent.log_prefix}      3. Use a smaller / faster reasoning "
    +                            f"model if the task doesn't require deep thinking.",
    +                            force=True,
    +                        )
    +
                         logger.error(
                             "%sAPI call failed after %s retries. %s | provider=%s model=%s msgs=%s tokens=~%s",
                             agent.log_prefix, max_retries, _final_summary,
    @@ -3543,7 +3602,22 @@ def _perform_api_call(next_api_kwargs):
                                 _final_response += f"\n\n{_billing_guidance}"
                         else:
                             _final_response = f"API call failed after {max_retries} retries: {_final_summary}"
    -                    if _is_stream_drop:
    +                    if _is_thinking_timeout:
    +                        # Thinking-timeout guidance overrides the generic
    +                        # stream-drop guidance — the latter is wrong for
    +                        # this case (it suggests splitting large file
    +                        # writes, which isn't what happened).  See the
    +                        # reasoning-model override at
    +                        # agent/error_classifier.py:720-738 and the
    +                        # detection block above for context.
    +                        from agent.thinking_timeout_guidance import (
    +                            build_thinking_timeout_guidance,
    +                        )
    +                        _final_response += build_thinking_timeout_guidance(
    +                            provider=_provider,
    +                            model=_model,
    +                        )
    +                    elif _is_stream_drop:
                             _final_response += (
                                 "\n\nThe provider's stream connection keeps "
                                 "dropping — this often happens when generating "

  • agent/error_classifier.py

    @@ -717,6 +717,26 @@ def _result(reason: FailoverReason, **overrides) -> ClassifiedError:
     
         is_disconnect = any(p in error_msg for p in _SERVER_DISCONNECT_PATTERNS)
         if is_disconnect and not status_code:
    +        # Reasoning-model override: a transport disconnect on a reasoning
    +        # model is much more likely the upstream proxy idle-killing a
    +        # long thinking stream than a true context overflow — even on
    +        # large sessions.  The default disconnect+large-session routing
    +        # below would otherwise send the user into the compression
    +        # branch (should_compress=True) and silently delete
    +        # conversation history on a phantom context-length error.
    +        # Reasoning models have multi-minute thinking phases that
    +        # routinely exceed the cloud gateway's idle window (NVIDIA
    +        # NIM ~120s — first-party repro at NVIDIA/NemoClaw#4846;
    +        # OpenAI worker / Anthropic stream-idle similar).  The
    +        # per-reasoning-model stale-timeout floor in
    +        # agent/reasoning_timeouts.py raises the stale-detector
    +        # threshold to tolerate long thinking, so a true
    +        # transport-layer failure here is recoverable via the retry
    +        # path — not via context compression.  Reclassify as timeout.
    +        # (Part 1 of Fixes #52310.)
    +        from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +        if get_reasoning_stale_timeout_floor(model) is not None:
    +            return _result(FailoverReason.timeout, retryable=True)
             # Absolute token/message-count thresholds are only a proxy for smaller
             # context windows.  Large-context sessions can have hundreds of
             # messages while still being far below their actual token budget.

  • agent/reasoning_timeouts.py

    @@ -0,0 +1,216 @@
    +"""Per-reasoning-model stale-timeout floor for known reasoning models.
    +
    +Reasoning models (those that emit extended thinking blocks before their
    +first content token) routinely exceed Hermes's default chat-model
    +stale detectors:
    +
    +* Stream stale detector:   ``HERMES_STREAM_STALE_TIMEOUT``     default 180s
    +                           ``agent/chat_completion_helpers.py:2544``
    +* Non-stream stale detector: ``HERMES_API_CALL_STALE_TIMEOUT``  default 90s
    +                           ``run_agent.py:1140``
    +
    +For NVIDIA Nemotron 3 Ultra on the hosted NIM gateway the empirical
    +upstream idle kill is ~120s (first-party reproduction at
    +NVIDIA/NemoClaw#4846 — TTFB ~31s, stream dies at 120s). The same
    +failure mode exists on OpenAI o1/o3, Anthropic Opus 4.x thinking,
    +DeepSeek R1, Qwen QwQ, xAI Grok reasoning — every cloud reasoning
    +model hits upstream-proxies / load-balancers with idle timeouts
    +shorter than the model's thinking phase. Result: the stale detector
    +kills the connection mid-think, surfacing as
    +``BrokenPipeError``/``RemoteProtocolError`` on the next read.
    +
    +This module provides a floor that the existing stale-detector scaling
    +blocks consult via :func:`get_reasoning_stale_timeout_floor` and
    +apply as ``max(default, floor)``. It is a FLOOR:
    +
    +* Never overrides explicit user config (``providers.<id>.models.<model>.stale_timeout_seconds``
    +  or ``request_timeout_seconds`` already wins — this code never runs
    +  in that branch).
    +* Never lowers an existing threshold.
    +* Has zero effect on non-reasoning models — they are not in the
    +  allowlist and the resolver returns ``None``.
    +
    +Matching uses start-anchored regex on the slug-only component of
    +the model name (after stripping any aggregator prefix like
    +``openai/``, ``x-ai/``, ``anthropic/``).  The right-anchor matches
    +end-of-string or a ``-``/``.``/``_`` slug separator, so ``qwen3-235b``
    +matches the ``qwen3`` family entry (a future model slug would be
    +``qwen3-235b-instruct`` and would also match) but ``some-other-qwen3``
    +does NOT match ``qwen3`` (the ``-qwen3`` is not at start of slug).
    +
    +The ``o1`` case is the most delicate: a model named
    +``llama-4-70b-o1-preview`` is a hypothetical community derivative that
    +should NOT trigger the reasoning-model floor for the user (the user
    +chose a non-OpenAI model, not a reasoning model).  The start-of-slug
    +anchor naturally excludes this — the matched ``o1-preview`` is at
    +position 11 of the slug, not at position 0.  The previous substring-
    +with-trailing-hyphen design would have over-matched here, which is
    +why start-of-slug anchoring is the right shape.
    +
    +Fixes #52217.
    +"""
    +
    +from __future__ import annotations
    +
    +import re
    +from typing import Optional
    +
    +
    +# (slug, floor_seconds).  Each slug is matched as a discrete
    +# word-boundary component via the wrapper regex in ``_match_any``
    +# below.  Order is irrelevant — the first regex match wins.
    +_REASONING_STALE_TIMEOUT_FLOORS: tuple[tuple[str, int], ...] = (
    +    # NVIDIA Nemotron — reasoning models behind hosted NIM with
    +    # documented 60-180s upstream idle kill (NVIDIA/NemoClaw#4846:
    +    # 120s measured).
    +    ("nemotron-3-ultra", 600),
    +    ("nemotron-3-super", 600),
    +    ("nemotron-3-nano",  300),
    +    # DeepSeek — R1 reasoning model on hosted NIM / DeepSeek direct.
    +    ("deepseek-r1", 600),
    +    ("deepseek-reasoner", 600),
    +    # Qwen — QwQ reasoning + Qwen3 thinking variants.  QwQ-32B
    +    # preview is the stable slug; ``qwen3`` covers the family of
    +    # thinking-mode Qwen3 models (qwen3-235b-a22b, qwen3-32b, etc.)
    +    # without over-matching every Qwen3 instruct variant — the
    +    # right-anchor requires the slug to be at the start of the
    +    # remaining model name, so ``qwen3-235b-instruct`` (instruct is
    +    # NOT a thinking variant) would still match.  Acceptable
    +    # trade-off: instruct variants of qwen3 get the 180s floor
    +    # even though they don't reason.  The cost is a slightly longer
    +    # wait on a hung provider; the alternative (matching only
    +    # ``qwen3-.*-thinking``) breaks the moment NVIDIA or Alibaba
    +    # ships a slightly different naming shape.
    +    ("qwq-32b", 300),
    +    ("qwen3", 180),
    +    # OpenAI o-series — known multi-minute TTFB.  Each variant
    +    # enumerated explicitly so bare ``o1`` doesn't over-match
    +    # ``olmo-1`` or hypothetical future community derivatives.
    +    ("o1", 600),
    +    ("o1-mini", 600),
    +    ("o1-pro", 600),
    +    ("o1-preview", 600),
    +    ("o3", 600),
    +    ("o3-pro", 600),
    +    ("o3-mini", 300),
    +    ("o4-mini", 300),
    +    # Anthropic Claude 4.x thinking variants.  Anchored at
    +    # ``claude-opus-4`` so non-thinking Claude 3.x or future
    +    # non-reasoning Claude variants don't match.
    +    ("claude-opus-4", 240),
    +    ("claude-sonnet-4.5", 180),
    +    ("claude-sonnet-4.6", 180),
    +    # xAI Grok reasoning variants.  Explicit reasoning-only keys
    +    # plus one for the ``non-reasoning`` variant so users picking
    +    # the fast variant don't get the 300s floor.  Bare ``grok-3``,
    +    # ``grok-4`` etc. don't match — only the explicit reasoning /
    +    # non-reasoning pairs.
    +    ("grok-4-fast-reasoning", 300),
    +    ("grok-4.20-reasoning", 300),
    +    ("grok-4-fast-non-reasoning", 180),
    +)
    +
    +
    +# Pre-compile each pattern.  Wrapper = start-of-slug + slug + end-or-
    +# separator, where ``start-of-slug`` means start-of-string OR
    +# immediately after the last ``/`` (aggregator separator) and
    +# ``end-or-separator`` means end-of-string OR a ``-``/``.``/``_``.
    +#
    +# Why start-of-slug and not start-of-string: aggregator prefixes
    +# like ``openai/`` should not affect matching — the slug identity is
    +# the part after the last ``/``.  Stripping the aggregator prefix in
    +# :func:`get_reasoning_stale_timeout_floor` before regex matching
    +# gives the wrapper a clean start-of-string anchor.
    +#
    +# Why end-or-separator on the right: ``openai/o3-mini`` must match
    +# the ``o3-mini`` slug (the right anchor is end-of-string).  And
    +# ``openai/o3-mini-2025-01-31`` must also match ``o3-mini`` (the right
    +# anchor is the ``-`` separator).  But ``openai/o3-mini-fork`` should
    +# NOT match ``o3-mini`` if we wanted to exclude forks — though the
    +# pattern ``o3-mini-fork`` would be matched as a derivative anyway,
    +# so we accept that community forks inheriting the same prefix are
    +# treated as reasoning models (a reasonable default — the upstream
    +# gateway timing is the same).
    +_PATTERN_CACHE: dict[str, re.Pattern[str]] = {}
    +
    +
    +def _get_pattern(slug: str) -> re.Pattern[str]:
    +    compiled = _PATTERN_CACHE.get(slug)
    +    if compiled is None:
    +        compiled = re.compile(
    +            r"^"
    +            + re.escape(slug)
    +            + r"(?:$|[\-._])"
    +        )
    +        _PATTERN_CACHE[slug] = compiled
    +    return compiled
    +
    +
    +def _match_any(model_lower: str) -> Optional[float]:
    +    """Return the floor for the first matching slug, else None.
    +
    +    Each table entry is matched as a start-of-slug prefix with the
    +    slug-separator-or-end-of-string right-anchor.  Table iteration
    +    order is irrelevant: longest slug wins (so ``o3-mini`` beats
    +    ``o3`` on a model like ``openai/o3-mini``).
    +    """
    +    # Sort by slug length descending so longer / more-specific slugs
    +    # win on shared prefixes (o3-mini beats o3).
    +    sorted_floors = sorted(
    +        _REASONING_STALE_TIMEOUT_FLOORS, key=lambda kv: -len(kv[0])
    +    )
    +    for slug, floor in sorted_floors:
    +        if _get_pattern(slug).search(model_lower):
    +            return float(floor)
    +    return None
    +
    +
    +def get_reasoning_stale_timeout_floor(model: object) -> Optional[float]:
    +    """Return the stale-timeout floor (seconds) for a known reasoning model.
    +
    +    Returns ``None`` when the model is not in the allowlist or the
    +    argument is empty / not a string.  Matching uses
    +    word-boundary-anchored regex on the lowercased model name, so
    +    ``openai/o3-mini`` matches the ``o3-mini`` slug but
    +    ``olmo-1`` does NOT match ``o1`` (the ``o1`` substring is not
    +    at a word boundary inside ``olmo-1``).
    +
    +    Aggregator prefixes (``openai/``, ``x-ai/``, ``anthropic/`` etc.)
    +    are preserved through matching — the ``/`` is itself a word
    +    boundary, so ``openai/o3-mini`` matches ``o3-mini`` because the
    +    ``/`` before ``o3-mini`` satisfies the left-anchor alternation.
    +
    +    This is a FLOOR — callers must apply it as ``max(default, floor)``
    +    and only when no explicit user-configured per-model
    +    ``stale_timeout_seconds`` exists.
    +
    +    >>> get_reasoning_stale_timeout_floor("nvidia/nemotron-3-ultra-550b-a55b")
    +    600.0
    +    >>> get_reasoning_stale_timeout_floor("openai/o3-mini")
    +    300.0
    +    >>> get_reasoning_stale_timeout_floor("deepseek/deepseek-r1")
    +    600.0
    +    >>> get_reasoning_stale_timeout_floor("qwen/qwen3-235b-a22b-thinking")
    +    180.0
    +    >>> get_reasoning_stale_timeout_floor("x-ai/grok-4-fast-reasoning")
    +    300.0
    +    >>> get_reasoning_stale_timeout_floor("anthropic/claude-opus-4-6")
    +    240.0
    +    >>> get_reasoning_stale_timeout_floor("gpt-4o") is None
    +    True
    +    >>> get_reasoning_stale_timeout_floor("olmo-1") is None
    +    True
    +    >>> get_reasoning_stale_timeout_floor(None) is None
    +    True
    +    """
    +    if not model or not isinstance(model, str):
    +        return None
    +    name = model.strip().lower()
    +    if not name:
    +        return None
    +    # Strip aggregator prefix (everything before and including the
    +    # last ``/``).  The wrapper regex anchors at start-of-string, so
    +    # the slug identity is the bare model name.
    +    if "/" in name:
    +        name = name.rsplit("/", 1)[1]
    +    return _match_any(name)

  • agent/thinking_timeout_guidance.py

    @@ -0,0 +1,136 @@
    +"""Thinking-timeout detection and user-facing guidance for reasoning models.
    +
    +When a known reasoning model (NVIDIA Nemotron 3 Ultra, OpenAI o1/o3,
    +Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning)
    +hits a transport-layer error before the first content token arrives, the
    +upstream proxy has almost certainly idle-killed a long thinking stream —
    +not a true context overflow or a configuration error.  The user needs
    +distinct guidance for this case:
    +
    +    "The model's thinking phase exceeded the upstream proxy's idle
    +     timeout before the first content token arrived.  This is a known
    +     issue with reasoning models behind cloud gateways (NVIDIA NIM,
    +     OpenAI, Anthropic, DeepSeek).  Workarounds in priority order:
    +     1. Set `providers.<provider>.models.<model>.stale_timeout_seconds: 900`
    +        in `~/.hermes/config.yaml` to extend the per-call timeout...
    +     2. Lower `reasoning_budget` or set `reasoning_effort: medium`...
    +     3. Use a smaller / faster reasoning model..."
    +
    +The existing `_is_stream_drop` guidance at
    +``agent/conversation_loop.py:3464-3486`` fires for large-file-write
    +stream drops ("try execute_code with Python's open() for large files")
    +which is the WRONG advice for the thinking-timeout case.  This module
    +provides the detection and the message as standalone helpers so the
    +detection logic is unit-testable without driving the full retry loop,
    +and the message text can be regression-tested for spelling and accuracy.
    +
    +Part 2 of Fixes #52310.
    +"""
    +
    +from __future__ import annotations
    +
    +from typing import Optional
    +
    +
    +# Substring set that identifies a transport-layer failure on the
    +# response stream.  Same shape as the existing
    +# ``_SERVER_DISCONNECT_PATTERNS`` in ``agent/error_classifier.py:394``
    +# but extended to also catch the OSS-level error signature
    +# (``broken pipe`` / ``errno 32``) that the upstream kill surfaces
    +# to the OpenAI SDK wrapper.
    +_THINKING_TIMEOUT_SUBSTRINGS: tuple[str, ...] = (
    +    "broken pipe",
    +    "errno 32",
    +    "remote protocol",
    +    "connection reset",
    +    "connection lost",
    +    "peer closed",
    +    "server disconnected",
    +)
    +
    +
    +def is_thinking_timeout(classified: object, model: str, error_msg: str) -> bool:
    +    """Return True when a reasoning model's thinking phase hit a transport kill.
    +
    +    Args:
    +        classified: a :class:`agent.error_classifier.ClassifiedError` instance
    +            (duck-typed here to avoid an import cycle in unit tests).
    +        model: the model slug at failure time (e.g.
    +            ``"nvidia/nemotron-3-ultra-550b-a55b"``).
    +        error_msg: lowercased string representation of the underlying
    +            exception (typically ``str(api_error).lower()``).
    +
    +    Returns True when ALL conditions hold:
    +        1. ``classified.reason == FailoverReason.timeout`` (the classifier
    +           override at ``agent/error_classifier.py:720-738`` ensures this
    +           is the case for reasoning models even on large sessions).
    +        2. ``api_error`` has no ``.status_code`` attribute set (transport
    +           disconnect, not an HTTP error).
    +        3. ``model`` is in the reasoning-model allowlist (reuses
    +           ``agent.reasoning_timeouts.get_reasoning_stale_timeout_floor``).
    +        4. ``error_msg`` contains one of the transport-kill substrings.
    +
    +    Non-reasoning models always return False.  Non-transport errors
    +    (billing / rate_limit / auth / context_overflow / format_error)
    +    always return False.  HTTP-status errors always return False.
    +    """
    +    # Import here (not at module top) to keep this helper cheap to
    +    # import even from callers that don't need it.  ``agent.reasoning_timeouts``
    +    # is small and dependency-free.
    +    from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +
    +    # Condition 1: classifier says timeout.  Use a string/value check
    +    # rather than importing FailoverReason so this module has zero
    +    # import cycles from the error_classifier package.
    +    reason = getattr(classified, "reason", None)
    +    reason_value = getattr(reason, "value", None)
    +    if reason_value != "timeout":
    +        return False
    +
    +    # Condition 2: no HTTP status code (transport, not API error).
    +    # Caller is expected to gate on ``getattr(api_error, "status_code", None) is None``
    +    # before calling this helper; the surface here is just the post-gate
    +    # boolean so the caller can pass an already-prepped error_msg.
    +
    +    # Condition 3: reasoning model allowlist.
    +    if get_reasoning_stale_timeout_floor(model) is None:
    +        return False
    +
    +    # Condition 4: transport-kill substring in the error message.
    +    error_msg_lower = (error_msg or "").lower()
    +    return any(p in error_msg_lower for p in _THINKING_TIMEOUT_SUBSTRINGS)
    +
    +
    +def build_thinking_timeout_guidance(
    +    provider: str, model: str, model_label: Optional[str] = None,
    +) -> str:
    +    """Return the user-facing guidance string appended to ``_final_response``.
    +
    +    Args:
    +        provider: provider slug (e.g. ``"nvidia"``, ``"openai"``).
    +        model: bare model slug the user would put in their config
    +            (e.g. ``"nemotron-3-ultra-550b-a55b"`` if the user uses
    +            NVIDIA direct, or the full ``"nvidia/nemotron-3-ultra-550b-a55b"``
    +            if they go through an aggregator).  Used verbatim in the
    +            config snippet so the user can copy-paste.
    +        model_label: optional short label for the model name in the
    +            prose (e.g. ``"Nemotron 3 Ultra"``).  Falls back to the
    +            slug if not provided.
    +    """
    +    label = model_label or model
    +    return (
    +        "\n\nThe model's thinking phase exceeded the upstream proxy's "
    +        "idle timeout before the first content token arrived. This is a "
    +        f"known issue with reasoning models (like {label}) behind cloud "
    +        "gateways (NVIDIA NIM, OpenAI, Anthropic, DeepSeek). Workarounds "
    +        "in priority order:\n"
    +        f"1. Set `providers.{provider}.models.{model}.stale_timeout_seconds: 900` "
    +        "in `~/.hermes/config.yaml` to extend the per-call timeout. "
    +        "(Hermes's built-in floor is 600s for known reasoning models — "
    +        "if you still see this after raising, the upstream cap is even "
    +        "shorter.)\n"
    +        "2. Lower `reasoning_budget` or set `reasoning_effort: medium` on this "
    +        "model if the provider supports it.\n"
    +        "3. Use a smaller / faster reasoning model if the task doesn't "
    +        "require deep thinking."
    +    )

  • scripts/release.py

    @@ -46,6 +46,7 @@
     # Auto-extracted from noreply emails + manual overrides
     AUTHOR_MAP = {
         "[redacted:email]": "agt-user",  # PR #48496 salvage (telegram CLOSE-WAIT polling heartbeat, #48495)
    +    "[redacted:email]": "DavidMetcalfe",  # PR #52272 salvage (route reasoning-model thinking-timeouts to timeout not context_overflow + reasoning-specific guidance; #52271)
         "[redacted:email]": "herbalizer404",  # PR #49076 + #51835 salvage (auxiliary compression fallback: 403/session-usage payment errors + honor fallback chain when aux provider auth unavailable)
         "[redacted:email]": "pyxl-dev",  # PR #52230 salvage (include rate-limit in auxiliary capacity-error fallback gate; #52228)
         "[redacted:email]": "benbenlijie",  # PR #47205 salvage (named custom-provider extra_body + Z.AI Coding overload adaptive backoff; #50663)

  • tests/agent/test_thinking_timeout_guidance.py

    @@ -0,0 +1,355 @@
    +"""Tests for the reasoning-model thinking-timeout detection + guidance.
    +
    +Two layers:
    +
    +1. **Classifier override (Part 1, ``agent/error_classifier.py:720-738``)**:
    +   A transport disconnect on a reasoning model is reclassified as
    +   ``FailoverReason.timeout`` even when the session is large — instead
    +   of routing to the compression branch via
    +   ``FailoverReason.context_overflow`` which would silently delete
    +   conversation history on a phantom context-length error.
    +
    +2. **Detection + guidance (Part 2, ``agent/thinking_timeout_guidance.py``)**:
    +   When the classifier says timeout AND the model is in the reasoning
    +   allowlist AND the error message has a transport-kill signature,
    +   the user gets actionable guidance (raise stale_timeout, lower
    +   reasoning_budget, or switch models) instead of the misleading
    +   "use execute_code with Python's open() for large files" advice
    +   that fires for the unrelated large-file-write stream-drop case.
    +
    +Both behaviors were previously broken: the existing
    +``test_disconnect_large_session_context_overflow`` test in
    +``tests/agent/test_error_classifier.py`` confirms that non-reasoning
    +models still route to context_overflow on a large session, so the
    +reasoning-model override is strictly targeted.
    +"""
    +
    +from __future__ import annotations
    +
    +from types import SimpleNamespace
    +
    +import pytest
    +
    +
    +# ── helpers ──────────────────────────────────────────────────────────────
    +
    +
    +class _TimeoutReason:
    +    """Minimal FailoverReason stand-in for unit tests."""
    +
    +    def __init__(self, value: str = "timeout") -> None:
    +        self.value = value
    +
    +
    +def _classified(reason: str = "timeout", **kwargs) -> SimpleNamespace:
    +    """Construct a ClassifiedError stand-in with the given reason."""
    +    defaults = dict(
    +        reason=_TimeoutReason(reason),
    +        status_code=None,
    +        retryable=True,
    +        should_compress=False,
    +        should_rotate_credential=[redacted:secret-assignment]
    +        should_fallback=False,
    +    )
    +    defaults.update(kwargs)
    +    return SimpleNamespace(**defaults)
    +
    +
    +# ── Part 1: classifier override (agent/error_classifier.py:720-738) ──
    +
    +
    +def _make_session(disconnect_message: str, model: str, *, num_messages: int = 250):
    +    """Construct inputs to classify_api_error for a disconnect+large-session case."""
    +    e = Exception(disconnect_message)
    +    # 128k context_length; 130k approx_tokens puts us over 0.6 of context
    +    # AND > 120k absolute threshold; 250 messages is also > 200 threshold.
    +    # Without the reasoning-model override, this routes to context_overflow.
    +    return e, {
    +        "provider": "nvidia",
    +        "model": model,
    +        "approx_tokens": 130_000,
    +        "context_length": 200_000,
    +        "num_messages": num_messages,
    +    }
    +
    +
    +class TestClassifierOverride:
    +    """The reasoning-model override at error_classifier.py:720-738.
    +
    +    Verifies the new behavior: a transport disconnect on a reasoning
    +    model on a LARGE session now routes to FailoverReason.timeout
    +    instead of context_overflow.  Without this fix, the compression
    +    branch would fire on a phantom overflow and silently delete
    +    conversation history.
    +    """
    +
    +    def test_reasoning_model_disconnect_on_large_session_is_timeout(self):
    +        from agent.error_classifier import classify_api_error, FailoverReason
    +        e, kwargs = _make_session(
    +            "server disconnected without sending complete message",
    +            model="nvidia/nemotron-3-ultra-550b-a55b",
    +        )
    +        result = classify_api_error(e, **kwargs)
    +        assert result.reason == FailoverReason.timeout, (
    +            "Reasoning-model transport disconnect on a large session "
    +            "should route to FailoverReason.timeout (not "
    +            "context_overflow) — the upstream proxy idle-kill is far "
    +            "more likely than a true context-length error on a "
    +            "thinking model."
    +        )
    +        assert result.should_compress is False, (
    +            "Compression would silently delete conversation history on "
    +            "a phantom overflow — must not fire for reasoning models."
    +        )
    +
    +    @pytest.mark.parametrize("model", [
    +        "nvidia/nemotron-3-ultra-550b-a55b",
    +        "openai/o3-mini",
    +        "anthropic/claude-opus-4-6",
    +        "deepseek/deepseek-r1",
    +        "qwen/qwq-32b-preview",
    +        "x-ai/grok-4-fast-reasoning",
    +    ])
    +    def test_all_known_reasoning_models_override(self, model):
    +        from agent.error_classifier import classify_api_error, FailoverReason
    +        e, kwargs = _make_session(
    +            "server disconnected without sending complete message",
    +            model=model,
    +        )
    +        result = classify_api_error(e, **kwargs)
    +        assert result.reason == FailoverReason.timeout
    +        assert result.should_compress is False
    +
    +    def test_non_reasoning_model_large_session_still_routes_to_context_overflow(self):
    +        """Regression guard: existing test_disconnect_large_session_context_overflow
    +        behavior must be preserved for non-reasoning models.
    +
    +        Without the override, this case routes to context_overflow +
    +        should_compress=True (the existing, intentional behavior for
    +        chat models that hit true context-length errors via proxy
    +        disconnect).  With the override, it stays that way.
    +        """
    +        from agent.error_classifier import classify_api_error, FailoverReason
    +        e, kwargs = _make_session(
    +            "server disconnected without sending complete message",
    +            model="gpt-4o",
    +        )
    +        result = classify_api_error(e, **kwargs)
    +        assert result.reason == FailoverReason.context_overflow
    +        assert result.should_compress is True
    +
    +    @pytest.mark.parametrize("model", [
    +        "olmo-1",
    +        "gpt-4o",
    +        "claude-3-5-sonnet-20240620",
    +        "llama-3.3-70b-instruct",
    +        "qwen2-72b-instruct",
    +        "x-ai/grok-3",
    +    ])
    +    def test_non_reasoning_models_all_keep_context_overflow(self, model):
    +        from agent.error_classifier import classify_api_error, FailoverReason
    +        e, kwargs = _make_session(
    +            "server disconnected without sending complete message",
    +            model=model,
    +        )
    +        result = classify_api_error(e, **kwargs)
    +        assert result.reason == FailoverReason.context_overflow
    +
    +    def test_reasoning_model_small_session_still_routes_to_timeout(self):
    +        """Sanity check: a reasoning model with a SMALL session also
    +        routes to timeout (the original behavior, unchanged by the
    +        override since the override's result matches the small-session
    +        branch's result)."""
    +        from agent.error_classifier import classify_api_error, FailoverReason
    +        e = Exception("server disconnected")
    +        result = classify_api_error(
    +            e,
    +            model="nvidia/nemotron-3-ultra-550b-a55b",
    +            approx_tokens=[redacted:secret-assignment]
    +            context_length=200_000,
    +            num_messages=10,
    +        )
    +        assert result.reason == FailoverReason.timeout
    +
    +    def test_reasoning_model_with_status_code_does_not_match_disconnect_pattern(self):
    +        """Status-code errors take the HTTP-status path in the
    +        classifier, not the disconnect-with-large-session path.
    +        The reasoning-model override is INSIDE the disconnect branch
    +        and doesn't fire for HTTP errors."""
    +        from agent.error_classifier import classify_api_error, FailoverReason
    +        e = Exception("server disconnected")
    +        # Inject a status_code attribute to simulate an HTTP error
    +        # whose message happens to contain "server disconnected".
    +        e.status_code = 503
    +        result = classify_api_error(
    +            e,
    +            model="nvidia/nemotron-3-ultra-550b-a55b",
    +            approx_tokens=[redacted:secret-assignment]
    +            context_length=200_000,
    +            num_messages=250,
    +        )
    +        # 503 specifically routes to overloaded (per the 5xx → 503/529
    +        # handling in error_classifier.py). The key assertion is that
    +        # the reasoning-model override is NOT reached — neither
    +        # timeout nor context_overflow.
    +        assert result.reason != FailoverReason.timeout
    +        assert result.reason != FailoverReason.context_overflow
    +        assert result.should_compress is False
    +
    +
    +# ── Part 2: detection (agent/thinking_timeout_guidance.py:is_thinking_timeout) ──
    +
    +
    +class TestIsThinkingTimeout:
    +    def test_returns_true_for_reasoning_model_with_transport_signature(self):
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason="timeout")
    +        assert is_thinking_timeout(
    +            classified,
    +            "nvidia/nemotron-3-ultra-550b-a55b",
    +            "Error communicating with OpenAI: [Errno 32] Broken pipe",
    +        ) is True
    +
    +    @pytest.mark.parametrize("model,msg", [
    +        ("nvidia/nemotron-3-ultra-550b-a55b", "connection reset by peer"),
    +        ("openai/o3-mini", "remote protocol error"),
    +        ("anthropic/claude-opus-4-6", "peer closed connection"),
    +        ("deepseek/deepseek-r1", "connection lost"),
    +        ("x-ai/grok-4-fast-reasoning", "server disconnected"),
    +    ])
    +    def test_known_reasoning_models_match(self, model, msg):
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason="timeout")
    +        assert is_thinking_timeout(classified, model, msg) is True
    +
    +    @pytest.mark.parametrize("model", [
    +        "gpt-4o",
    +        "claude-3-5-sonnet-20240620",
    +        "llama-3.3-70b-instruct",
    +        "qwen2-72b-instruct",
    +    ])
    +    def test_non_reasoning_models_never_match(self, model):
    +        """Non-reasoning models must always return False even with
    +        matching transport signature — the guidance is
    +        reasoning-specific."""
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason="timeout")
    +        assert is_thinking_timeout(
    +            classified, model, "connection reset by peer",
    +        ) is False
    +
    +    @pytest.mark.parametrize("reason", [
    +        "billing",
    +        "rate_limit",
    +        "auth",
    +        "context_overflow",
    +        "format_error",
    +        "provider_policy_blocked",
    +        "content_policy_blocked",
    +        "thinking_signature",
    +        "unknown",
    +    ])
    +    def test_non_timeout_reasons_never_match(self, reason):
    +        """The detection only fires when the classifier says timeout.
    +        Other reasons have their own distinct guidance paths."""
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason=reason)
    +        assert is_thinking_timeout(
    +            classified,
    +            "nvidia/nemotron-3-ultra-550b-a55b",
    +            "connection reset by peer",
    +        ) is False
    +
    +    @pytest.mark.parametrize("msg", [
    +        "Insufficient credits",
    +        "Rate limit exceeded",
    +        "Invalid API key",
    +        "Context length exceeded",
    +        "Tool call argument malformed",
    +    ])
    +    def test_non_transport_messages_never_match(self, msg):
    +        """The detection only fires for transport-kill signatures.
    +        A reasoning model that returns a billing/rate-limit/auth/etc
    +        error gets the classifier-default guidance, not this one."""
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason="timeout")
    +        assert is_thinking_timeout(
    +            classified, "nvidia/nemotron-3-ultra-550b-a55b", msg,
    +        ) is False
    +
    +    def test_empty_error_msg_returns_false(self):
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason="timeout")
    +        assert is_thinking_timeout(
    +            classified, "nvidia/nemotron-3-ultra-550b-a55b", "",
    +        ) is False
    +
    +    def test_none_error_msg_returns_false(self):
    +        from agent.thinking_timeout_guidance import is_thinking_timeout
    +        classified = _classified(reason="timeout")
    +        assert is_thinking_timeout(
    +            classified, "nvidia/nemotron-3-ultra-550b-a55b", None,
    +        ) is False
    +
    +
    +# ── Part 2: guidance text (agent/thinking_timeout_guidance.py:build_thinking_timeout_guidance) ──
    +
    +
    +class TestBuildThinkingTimeoutGuidance:
    +    def test_guidance_mentions_config_path(self):
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(
    +            provider="nvidia", model="nvidia/nemotron-3-ultra-550b-a55b",
    +        )
    +        assert "providers.nvidia.models.nvidia/nemotron-3-ultra-550b-a55b.stale_timeout_seconds" in text
    +
    +    def test_guidance_mentions_three_workarounds(self):
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(provider="nvidia", model="x")
    +        assert "1." in text
    +        assert "2." in text
    +        assert "3." in text
    +
    +    def test_guidance_mentions_known_providers(self):
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(provider="nvidia", model="x")
    +        # At least one of the known cloud providers should be mentioned
    +        # to give the user context.
    +        assert any(p in text for p in (
    +            "NVIDIA NIM", "OpenAI", "Anthropic", "DeepSeek",
    +        ))
    +
    +    def test_guidance_mentions_built_in_floor(self):
    +        """User should know that 600s is already set by default for
    +        known reasoning models — if they hit the error after raising,
    +        it's the upstream cap, not hermes."""
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(provider="nvidia", model="x")
    +        assert "600s" in text
    +
    +    def test_guidance_does_not_recommend_execute_code(self):
    +        """Critical regression guard: the new guidance must NOT
    +        recommend `execute_code with Python's open() for large files`
    +        — that's the misleading advice from the existing _is_stream_drop
    +        guidance that was wrong for the thinking-timeout case."""
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(provider="nvidia", model="x")
    +        assert "execute_code" not in text
    +        assert "open()" not in text
    +
    +    def test_guidance_uses_label_when_provided(self):
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(
    +            provider="nvidia",
    +            model="nvidia/nemotron-3-ultra-550b-a55b",
    +            model_label="Nemotron 3 Ultra",
    +        )
    +        assert "Nemotron 3 Ultra" in text
    +
    +    def test_guidance_falls_back_to_slug_when_no_label(self):
    +        from agent.thinking_timeout_guidance import build_thinking_timeout_guidance
    +        text = build_thinking_timeout_guidance(
    +            provider="nvidia",
    +            model="nvidia/nemotron-3-ultra-550b-a55b",
    +        )
    +        assert "nvidia/nemotron-3-ultra-550b-a55b" in text

• #52845 feat(agent): apply per-reasoning-model stale-timeout floor in stream + non-stream detectors (#52217)

  merged to main

  files: agent/chat_completion_helpers.py, run_agent.py, tests/agent/test_reasoning_stale_timeout_floor.py

  Show the diff

  • agent/chat_completion_helpers.py

    @@ -2561,6 +2561,17 @@ def _call():
                 _stream_stale_timeout = max(_stream_stale_timeout_base, 240.0)
             else:
                 _stream_stale_timeout = _stream_stale_timeout_base
    +        # Reasoning-model floor: known reasoning models (Nemotron 3 Ultra,
    +        # OpenAI o1/o3, Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ,
    +        # xAI Grok reasoning, etc.) routinely exceed the default 180s chat-
    +        # model threshold during their thinking phase.  The cloud gateway
    +        # upstream kills the socket first, surfacing as BrokenPipeError.
    +        # Raises the floor only — never overrides explicit user config
    +        # (handled by get_provider_stale_timeout above).
    +        from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +        _reasoning_floor = get_reasoning_stale_timeout_floor(api_kwargs.get("model"))
    +        if _reasoning_floor is not None:
    +            _stream_stale_timeout = max(_stream_stale_timeout, _reasoning_floor)
     
         t = threading.Thread(target=_call, daemon=True)
         t.start()

  • run_agent.py

    @@ -1138,6 +1138,19 @@ def _resolved_api_call_stale_timeout_base(self) -> tuple[float, bool]:
             if env_timeout is not None:
                 return float(env_timeout), False
     
    +        # Reasoning-model floor: auto-mitigation for known reasoning models
    +        # (Nemotron 3 Ultra, OpenAI o1/o3, Anthropic Opus 4.x thinking,
    +        # DeepSeek R1, Qwen QwQ, xAI Grok reasoning, etc.) whose cloud
    +        # gateways idle-kill before the model's thinking phase ends.
    +        # uses_implicit_default is False here so the local-endpoint
    +        # short-circuit in _compute_non_stream_stale_timeout does not
    +        # disable stale detection for users running reasoning models on a
    +        # local NIM endpoint.
    +        from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +        reasoning_floor = get_reasoning_stale_timeout_floor(self.model)
    +        if reasoning_floor is not None:
    +            return reasoning_floor, False
    +
             return 90.0, True
     
         def _compute_non_stream_stale_timeout(self, api_payload: Any) -> float:

  • tests/agent/test_reasoning_stale_timeout_floor.py

    @@ -0,0 +1,373 @@
    +"""Regression tests for the reasoning-model stale-timeout floor (issue #52217).
    +
    +Reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Anthropic Opus 4.x
    +thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning) routinely exceed
    +the 180s / 90s chat-model stale-timeout defaults during their
    +thinking phase.  Hermes's default cloud-stream stale detector
    +(``HERMES_STREAM_STALE_TIMEOUT`` = 180s) and non-stream detector
    +(``HERMES_API_CALL_STALE_TIMEOUT`` = 90s) both fire before the
    +upstream proxy's idle timeout on a healthy reasoning stream.  Result:
    +the user sees ``API call failed after 3 retries: [Errno 32] Broken
    +pipe`` for every Nemotron 3 Ultra turn.
    +
    +These tests pin the floor's behavior:
    +
    +1. ``get_reasoning_stale_timeout_floor`` returns the right floor for
    +   every key in the allowlist, ``None`` for every negative case
    +   (gpt-4o, olmo-1, etc.), and longest-substring-first wins on
    +   shared prefixes (``o3-mini-`` > ``o3-``).
    +2. The non-stream resolver at
    +   ``run_agent.py:AIAgent._resolved_api_call_stale_timeout_base``
    +   consults the floor at priority 4 (after explicit user config,
    +   provider config, and env var; before the 90s default), and
    +   returns ``uses_implicit_default=False`` so the local-endpoint
    +   short-circuit in ``_compute_non_stream_stale_timeout`` does not
    +   disable stale detection for a reasoning model running on a local
    +   NIM endpoint.
    +3. The stream stale-timeout resolution (mirrored here as in
    +   ``test_stream_read_timeout_floor.py`` because the real builder
    +   lives inside a worker thread) consults the floor after the
    +   context-size scaling block, raising the timeout for reasoning
    +   models without lowering it for non-reasoning models.
    +"""
    +
    +from __future__ import annotations
    +
    +from pathlib import Path
    +
    +import pytest
    +
    +
    +# ── pure-function resolver ────────────────────────────────────────────────
    +
    +
    [redacted:email]("model,expected", [
    +    # NVIDIA Nemotron reasoning family (longest keys first).
    +    ("nvidia/nemotron-3-ultra-550b-a55b", 600.0),
    +    ("nvidia/nemotron-3-super-120b-a12b", 600.0),
    +    ("nvidia/nemotron-3-nano-30b-a3b", 300.0),
    +    # DeepSeek R1 + DeepSeek reasoner.
    +    ("deepseek/deepseek-r1", 600.0),
    +    ("deepseek/deepseek-r1-distill-llama-70b", 600.0),
    +    ("deepseek/deepseek-reasoner", 600.0),
    +    # Qwen QwQ + Qwen3 thinking variants (qwen3 family entry matches all).
    +    ("qwen/qwq-32b-preview", 300.0),
    +    ("qwen/qwen3-235b-a22b-thinking", 180.0),
    +    ("qwen/qwen3-32b", 180.0),
    +    # OpenAI o-series — each variant enumerated explicitly.
    +    # Longest match wins (o3-mini beats o3 on shared prefix).
    +    ("openai/o1", 600.0),
    +    ("openai/o1-mini", 600.0),
    +    ("openai/o1-pro", 600.0),
    +    ("openai/o1-preview", 600.0),
    +    ("openai/o3", 600.0),
    +    ("openai/o3-pro", 600.0),
    +    ("openai/o3-mini", 300.0),
    +    ("openai/o4-mini", 300.0),
    +    # Anthropic Claude 4.x thinking variants.
    +    ("anthropic/claude-opus-4-6", 240.0),
    +    ("anthropic/claude-opus-4-20250514", 240.0),
    +    ("anthropic/claude-sonnet-4.5", 180.0),
    +    ("anthropic/claude-sonnet-4.6", 180.0),
    +    # xAI Grok reasoning variants — explicit, not bare `grok`.
    +    ("x-ai/grok-4-fast-reasoning", 300.0),
    +    ("x-ai/grok-4.20-reasoning", 300.0),
    +    ("x-ai/grok-4-fast-non-reasoning", 180.0),
    +])
    +def test_reasoning_stale_timeout_floor_positive_cases(model, expected):
    +    from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +    assert get_reasoning_stale_timeout_floor(model) == expected, (
    +        f"get_reasoning_stale_timeout_floor({model!r}) should return "
    +        f"{expected}; bare substrings and shared prefixes must not "
    +        f"over-match community derivatives."
    +    )
    +
    +
    [redacted:email]("model", [
    +    # Non-reasoning chat models — no floor.
    +    "gpt-4o",
    +    "gpt-5",
    +    "claude-3-5-sonnet-20240620",
    +    "llama-3.3-70b-instruct",
    +    "gemini-2.5-pro",
    +    # Start-of-slug anchor traps — the slug must be at the START of
    +    # the bare model name (after aggregator-prefix strip).  Bare
    +    # substring matching would over-match these.
    +    "olmo-1",
    +    "olmo-13b",
    +    "llama-4-70b-o1-preview",     # embedded `o1-preview`, NOT start of slug
    +    "some-model-o3-mini-fork",    # embedded `o3-mini`, NOT start of slug
    +    # Bare "grok" must not over-match non-reasoning Grok SKUs.
    +    "x-ai/grok-3",
    +    "x-ai/grok-4",
    +    "x-ai/grok-4-0709",
    +    "x-ai/grok-code-fast-1",
    +    # Qwen2 must not match Qwen3 (different family).
    +    "qwen2-72b-instruct",
    +    # Empty / None / non-string inputs — must return None, not raise.
    +    "",
    +    None,
    +    12345,
    +    [],
    +])
    +def test_reasoning_stale_timeout_floor_negative_cases(model):
    +    from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +    assert get_reasoning_stale_timeout_floor(model) is None, (
    +        f"get_reasoning_stale_timeout_floor({model!r}) must return None "
    +        f"for non-reasoning models and start-of-slug-anchor traps."
    +    )
    +
    +
    +def test_longest_substring_wins_on_shared_prefix():
    +    """`o3-mini` must beat `o3` so the smaller floor applies."""
    +    from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +    # o3-mini (7 chars) wins over o3 (2 chars) on shared prefix.
    +    assert get_reasoning_stale_timeout_floor("openai/o3-mini") == 300.0
    +    assert get_reasoning_stale_timeout_floor("openai/o3") == 600.0
    +    # Even with deep aggregator prefix chains the model name resolves
    +    # correctly (start-of-slug anchor + rsplit('/') strip).
    +    assert get_reasoning_stale_timeout_floor("openrouter/openai/o3-mini") == 300.0
    +    assert get_reasoning_stale_timeout_floor("openrouter/anthropic/claude-opus-4-6") == 240.0
    +
    +
    +
    +# ── integration: _resolved_api_call_stale_timeout_base ─────────────────────
    +
    +
    +def _write_config(tmp_path: Path, body: str) -> None:
    +    (tmp_path / "config.yaml").write_text(body or "{}\n", encoding="utf-8")
    +
    +
    +def _make_agent(tmp_path: Path, **overrides):
    +    from run_agent import AIAgent
    +    kwargs = dict(
    +        model="gpt-5.5",
    +        provider="openai-codex",
    +        api_key="[redacted:secret-assignment]",
    +        base_url="https://chatgpt.com/backend-api/codex",
    +        quiet_mode=True,
    +        skip_context_files=True,
    +        skip_memory=True,
    +        platform="cli",
    +    )
    +    kwargs.update(overrides)
    +    return AIAgent(**kwargs)
    +
    +
    +def test_reasoning_floor_applies_to_nemotron_3_ultra(monkeypatch, tmp_path):
    +    """Nemotron 3 Ultra without explicit config gets the 600s floor."""
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    (tmp_path / ".env").write_text("", encoding="utf-8")
    +    monkeypatch.delenv("HERMES_API_CALL_STALE_TIMEOUT", raising=False)
    +    _write_config(tmp_path, "")
    +
    +    # Clear any cached config from prior tests in this session.
    +    import importlib
    +    from hermes_cli import config as cfg_mod, timeouts as to_mod
    +    importlib.reload(cfg_mod)
    +    importlib.reload(to_mod)
    +
    +    agent = _make_agent(
    +        tmp_path,
    +        provider="nvidia",
    +        base_url="https://integrate.api.nvidia.com/v1",
    +        model="nvidia/nemotron-3-ultra-550b-a55b",
    +    )
    +    base, implicit = agent._resolved_api_call_stale_timeout_base()
    +    assert base == 600.0
    +    assert implicit is False, (
    +        "Reasoning-model floor must return uses_implicit_default=False "
    +        "so the local-endpoint short-circuit in "
    +        "_compute_non_stream_stale_timeout does not disable detection "
    +        "for users running reasoning models on a local NIM endpoint."
    +    )
    +
    +
    +def test_reasoning_floor_applies_to_opus_4_thinking(monkeypatch, tmp_path):
    +    """Anthropic Opus 4.x thinking gets the 240s floor without explicit config."""
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    (tmp_path / ".env").write_text("", encoding="utf-8")
    +    monkeypatch.delenv("HERMES_API_CALL_STALE_TIMEOUT", raising=False)
    +    _write_config(tmp_path, "")
    +
    +    import importlib
    +    from hermes_cli import config as cfg_mod, timeouts as to_mod
    +    importlib.reload(cfg_mod)
    +    importlib.reload(to_mod)
    +
    +    agent = _make_agent(
    +        tmp_path,
    +        provider="anthropic",
    +        base_url="https://api.anthropic.com",
    +        model="claude-opus-4-6",
    +    )
    +    base, implicit = agent._resolved_api_call_stale_timeout_base()
    +    assert base == 240.0
    +    assert implicit is False
    +
    +
    +def test_reasoning_floor_never_overrides_explicit_user_config(monkeypatch, tmp_path):
    +    """Explicit per-model stale_timeout_seconds wins over the floor.
    +
    +    Regression guard for the invariant: explicit user config > reasoning
    +    floor > env var > default. If a user sets stale_timeout_seconds: 60
    +    on Nemotron 3 Ultra, that's what fires — even though the floor
    +    would otherwise be 600s.
    +    """
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    (tmp_path / ".env").write_text("", encoding="utf-8")
    +    _write_config(tmp_path, """\
    +providers:
    +  nvidia:
    +    models:
    +      nvidia/nemotron-3-ultra-550b-a55b:
    +        stale_timeout_seconds: 60
    +""")
    +    monkeypatch.delenv("HERMES_API_CALL_STALE_TIMEOUT", raising=False)
    +
    +    import importlib
    +    from hermes_cli import config as cfg_mod, timeouts as to_mod
    +    importlib.reload(cfg_mod)
    +    importlib.reload(to_mod)
    +
    +    agent = _make_agent(
    +        tmp_path,
    +        provider="nvidia",
    +        base_url="https://integrate.api.nvidia.com/v1",
    +        model="nvidia/nemotron-3-ultra-550b-a55b",
    +    )
    +    base, implicit = agent._resolved_api_call_stale_timeout_base()
    +    assert base == 60.0, (
    +        "Explicit user stale_timeout_seconds must override the "
    +        "reasoning-model floor; the user knows their environment."
    +    )
    +    assert implicit is False
    +
    +
    +def test_reasoning_floor_loses_to_env_var_when_no_floor_match(monkeypatch, tmp_path):
    +    """For a non-reasoning model, env var still wins over the 90s default."""
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    (tmp_path / ".env").write_text("", encoding="utf-8")
    +    monkeypatch.setenv("HERMES_API_CALL_STALE_TIMEOUT", "300")
    +    _write_config(tmp_path, "")
    +
    +    import importlib
    +    from hermes_cli import config as cfg_mod, timeouts as to_mod
    +    importlib.reload(cfg_mod)
    +    importlib.reload(to_mod)
    +
    +    agent = _make_agent(
    +        tmp_path,
    +        provider="openai",
    +        base_url="https://api.openai.com/v1",
    +        model="gpt-5.5",  # not in the floor allowlist
    +    )
    +    base, implicit = agent._resolved_api_call_stale_timeout_base()
    +    assert base == 300.0
    +    assert implicit is False
    +
    +
    +def test_non_reasoning_model_keeps_default(monkeypatch, tmp_path):
    +    """GPT-5 (non-reasoning) without env var / config -> 90s default, implicit."""
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    (tmp_path / ".env").write_text("", encoding="utf-8")
    +    monkeypatch.delenv("HERMES_API_CALL_STALE_TIMEOUT", raising=False)
    +    _write_config(tmp_path, "")
    +
    +    import importlib
    +    from hermes_cli import config as cfg_mod, timeouts as to_mod
    +    importlib.reload(cfg_mod)
    +    importlib.reload(to_mod)
    +
    +    agent = _make_agent(
    +        tmp_path,
    +        provider="openai",
    +        base_url="https://api.openai.com/v1",
    +        model="gpt-5.5",
    +    )
    +    base, implicit = agent._resolved_api_call_stale_timeout_base()
    +    assert base == 90.0
    +    assert implicit is True
    +
    +
    +# ── stream-side mirror (the real builder lives in a worker thread) ────────
    +
    +
    +def _resolve_stream_stale_timeout(
    +    model: str | None,
    +    base_url: str,
    +    est_tokens: int,
    +    stale_base: float = 180.0,
    +) -> float:
    +    """Mirror of the stale-stream resolution in agent/chat_completion_helpers.py.
    +
    +    Kept in lockstep with the production code at lines 2539-2575 of
    +    agent/chat_completion_helpers.py.  When that block changes, this
    +    mirror must change too — the failing-test signal is the divergence.
    +    """
    +    from agent.model_metadata import is_local_endpoint
    +    from agent.reasoning_timeouts import get_reasoning_stale_timeout_floor
    +
    +    # Provider-configured stale timeout wins (mirrors get_provider_stale_timeout).
    +    if stale_base != 180.0:
    +        pass  # In production this is sourced from config; here we parameterize.
    +
    +    if stale_base == 180.0 and base_url and is_local_endpoint(base_url):
    +        return float("inf")
    +
    +    if est_tokens > 100_000:
    +        timeout = max(stale_base, 300.0)
    +    elif est_tokens > 50_000:
    +        timeout = max(stale_base, 240.0)
    +    else:
    +        timeout = stale_base
    +
    +    # Reasoning-model floor (the new branch this PR adds).
    +    floor = get_reasoning_stale_timeout_floor(model)
    +    if floor is not None:
    +        timeout = max(timeout, floor)
    +    return timeout
    +
    +
    +def test_stream_stale_timeout_floor_for_nemotron_3_ultra():
    +    """Small-context Nemotron 3 Ultra without explicit config -> 600s floor.
    +
    +    Without the floor, this would be 180s (the default), which is shorter
    +    than NVIDIA NIM's ~120s upstream idle kill — guaranteeing broken pipe.
    +    """
    +    timeout = _resolve_stream_stale_timeout(
    +        model="nvidia/nemotron-3-ultra-550b-a55b",
    +        base_url="https://integrate.api.nvidia.com/v1",
    +        est_tokens=[redacted:secret-assignment]
    +    )
    +    assert timeout == 600.0
    +
    +
    +def test_stream_stale_timeout_floor_never_lowers_existing():
    +    """The floor raises; it never lowers the existing context-size tier."""
    +    # 120k-token conversation on a reasoning model -> context tier already
    +    # raises to 300s; floor (600s) takes it to 600s.
    +    timeout = _resolve_stream_stale_timeout(
    +        model="nvidia/nemotron-3-ultra-550b-a55b",
    +        base_url="https://integrate.api.nvidia.com/v1",
    +        est_tokens=[redacted:secret-assignment]
    +    )
    +    assert timeout == 600.0
    +
    +    # 60k tokens on Opus 4 -> context tier raises to 240s; floor keeps 240s.
    +    timeout = _resolve_stream_stale_timeout(
    +        model="anthropic/claude-opus-4-6",
    +        base_url="https://api.anthropic.com",
    +        est_tokens=[redacted:secret-assignment]
    +    )
    +    assert timeout == 240.0
    +
    +
    +def test_stream_stale_timeout_unchanged_for_non_reasoning_models():
    +    """gpt-4o on a small context still gets the 180s default — no behavior change."""
    +    timeout = _resolve_stream_stale_timeout(
    +        model="gpt-4o",
    +        base_url="https://api.openai.com/v1",
    +        est_tokens=[redacted:secret-assignment]
    +    )
    +    assert timeout == 180.0

• #121555 fix(agent): run-budget caps stream stale timeout; MiniMax M2.x reasoning floor; inline cron Codex keeps its floor

  merged to main

  files: agent/chat_completion_helpers.py, agent/reasoning_timeouts.py, contributors/emails/[redacted:email], contributors/emails/[redacted:email], run_agent.py, tests/agent/test_cron_inline_api_call.py, tests/agent/test_reasoning_stale_timeout_floor.py, tests/agent/test_run_budget.py, tests/cron/test_cron_direct_api_call.py, tests/cron/test_cron_direct_api_call_watchdog.py