# NousResearch/hermes-agent · Internal message fields leaking to strict API providers

## Open now

- 1 issue, 3 PRs open on GitHub.
- #48523: convert_messages doesn't strip timestamp/message_id/observed/finish_reason — causes 400 with strict providers in gateway mode
- PR #48170: fix(agent): strip timestamp metadata from chat payloads
- PR #48916: fix(prompt): honor explicit system prompt timestamp config
- PR #49595: fix(transport): strip timestamp and observed from chat completions messages

Multiple bug reports and fixes address internal bookkeeping fields (timestamp, message_id, observed, etc.) being included in outgoing API message payloads, causing HTTP 400/422 errors on strict-schema providers like Mistral and GLM. The fixes involve stripping these fields before sending requests to external LLM endpoints.

## TL;DR

- 29 member(s): 1 open issue(s), 10 closed issue(s), 3 open PR(s), 2 merged PR(s), 13 PR(s) closed without merging.
- A fix was merged to main in PR #31456. Verify it addresses your case.
- **Recommended:** read PR #31456 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-31456,pr-49840`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text.
- Members outside that structural core may belong to a different underlying problem.
- Pairwise link density `0.07`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Open related issues

- #48523: convert_messages doesn't strip timestamp/message_id/observed/finish_reason — causes 400 with strict providers in gateway mode

## Closed / duplicate issues

- #30959: Internal bookkeeping fields leak to strict providers, causing HTTP 400 retry loops
- #47868: Strict chat-completions providers reject leaked messages\[\].timestamp metadata
- #48263: \[Bug\]: GLM 5.2 via OpenCode Go fails with HTTP 400 — internal \`timestamp\` field leaked into API messages
- #48386: \[Bug\]: Gateway sends timestamp metadata field in message objects → HTTP 422 on strict OpenAI-compatible endpoints (api.mistral.ai)
- #48772: cannot disable the timestamp in config
- #49490: \[Bug\]: OpenCode Zen provider does not work on v0.17.0
- #49572: chat_completions transport: timestamp field on message dicts causes HTTP 400 on strict providers
- #53139: \[BUG\] GLM 5.2 multi-turn sessions fail via OpenCode Go due to timestamp field injection
- #55902: HTTP 400 with OpenCode Go + Kimi K2.5: 'messages\[N\].timestamp' is not permitted
- #68077: sanitize_api_messages() doesn't strip gateway-only timestamp/message_id keys, causing HTTP 400 on strict-schema providers (GLM-5.2)

## Prior & competing PRs

- **#31456: fix(transport): strip Hermes-internal scaffolding keys + classify 5xx validation errors (salvage #26054)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `agent/error_classifier.py`, `agent/transports/chat_completions.py`, `scripts/release.py`, `tests/agent/test_error_classifier.py`, `tests/agent/transports/test_chat_completions.py`
  - fix hunks:
  - `agent/error_classifier.py`

    ```diff
    @@ -240,6 +240,24 @@ def is_auth(self) -> bool:
         "unsupported model",
     ]
     
    +# Request-validation patterns — the request is malformed and will fail
    +# identically on every retry. Some OpenAI-compatible gateways (notably
    +# codex.nekos.me) return these as 5xx instead of the standard 4xx, which
    +# makes the generic "5xx → retryable server_error" rule misfire: the retry
    +# loop hammers the same deterministic rejection 3+ times, then the
    +# transport-recovery path resets the counter and does it again, producing
    +# a request flood. When a 5xx body carries one of these unambiguous
    +# request-validation signals, classify as a non-retryable format_error so
    +# the loop fails fast and falls back instead of looping.
    +_REQUEST_VALIDATION_PATTERNS = [
    +    "unknown parameter",
    +    "unsupported parameter",
    +    "unrecognized request argument",
    +    "invalid_request_error",
    +    "unknown_parameter",
    +    "unsupported_parameter",
    +]
    +
     # OpenRouter aggregator policy-block patterns.
     #
     # When a user's OpenRouter account privacy setting (or a per-request
    ```

    ```diff
    @@ -745,6 +763,23 @@ def _classify_by_status(
             )
     
         if status_code in {500, 502}:
    +        # Some OpenAI-compatible gateways return request-validation errors
    +        # with a 5xx status (codex.nekos.me returns 502 for unknown/
    +        # unsupported parameters). These are deterministic — every retry
    +        # gets the identical rejection — so the generic "5xx → retryable
    +        # server_error" rule turns one bad request into a retry flood.
    +        # Detect the unambiguous request-validation signals (in either the
    +        # message text or the structured error code) and fail fast.
    +        if (
    +            any(p in error_msg for p in _REQUEST_VALIDATION_PATTERNS)
    +            or error_code.lower() in {"invalid_request_error", "unknown_parameter",
    +                                      "unsupported_parameter"}
    +        ):
    +            return result_fn(
    +                FailoverReason.format_error,
    +                retryable=False,
    +                should_fallback=True,
    +            )
             return result_fn(FailoverReason.server_error, retryable=True)
     
         if status_code in {503, 529}:
    ```
  - `agent/transports/chat_completions.py`

    ```diff
    @@ -113,9 +113,8 @@ def convert_messages(
             self, messages: list[dict[str, Any]], **kwargs
         ) -> list[dict[str, Any]]:
             """Messages are already in OpenAI format — strip internal fields
    -        that strict chat-completions providers reject with HTTP 400/422.
    -
    -        Strips:
    +        that strict chat-completions providers reject with HTTP 400/422
    +        (or, in the case of some OpenAI-compatible gateways, 5xx):
     
             - Codex Responses API fields: ``codex_reasoning_items`` /
               ``codex_message_items`` on the message, ``call_id`` /
    ```

    ```diff
    @@ -127,6 +126,16 @@ def convert_messages(
               ``Extra inputs are not permitted, field: 'messages[N].tool_name'``.
               Permissive providers (OpenRouter, MiniMax) silently ignore the
               field, which masked the bug for months.
    +        - Hermes-internal scaffolding markers — any top-level message key
    +          starting with ``_`` (e.g. ``_empty_recovery_synthetic``,
    +          ``_empty_terminal_sentinel``, ``_thinking_prefill``). These are
    +          bookkeeping flags the agent loop attaches to messages so the
    +          persistence layer can later strip its own scaffolding; they must
    +          never reach the wire. Permissive providers (real OpenAI,
    +          Anthropic) silently drop unknown message keys, but strict
    +          gateways (e.g. opencode-go, codex.nekos.me) reject with
    +          ``Extra inputs are not permitted, field: 'messages[N]._empty_recovery_synthetic'``,
    +          which then poisons every subsequent request in the session.
             """
             needs_sanitize = False
             for msg in messages:
    ```

    ```diff
    @@ -139,6 +148,9 @@ def convert_messages(
                 ):
                     needs_sanitize = True
                     break
    +            if any(isinstance(k, str) and k.startswith("_") for k in msg):
    +                needs_sanitize = True
    +                break
                 tool_calls = msg.get("tool_calls")
                 if isinstance(tool_calls, list):
                     for tc in tool_calls:
    ```

    ```diff
    @@ -160,6 +172,11 @@ def convert_messages(
                 msg.pop("codex_reasoning_items", None)
                 msg.pop("codex_message_items", None)
                 msg.pop("tool_name", None)
    +            # Drop all Hermes-internal scaffolding markers (``_``-prefixed).
    +            # OpenAI's message schema has no ``_``-prefixed fields, so this
    +            # is safe and future-proofs against new markers being added.
    +            for key in [k for k in msg if isinstance(k, str) and k.startswith("_")]:
    +                msg.pop(key, None)
                 tool_calls = msg.get("tool_calls")
                 if isinstance(tool_calls, list):
                     for tc in tool_calls:
    ```
  - `scripts/release.py`

    ```diff
    @@ -70,6 +70,8 @@
         "[redacted:email]": "qiyin-code",
         "[redacted:email]": "aashizpoudel",
         "[redacted:email]": "shaun0927",
    +    "[redacted:email]": "Soju06",
    +    "[redacted:email]": "Soju06",
         "[redacted:email]": "Bihruze",
         "[redacted:email]": "Lempkey",
         "[redacted:email]": "m0n3r0",
    ```
  - `tests/agent/test_error_classifier.py`

    ```diff
    @@ -293,6 +293,64 @@ def test_529_anthropic_overloaded(self):
             result = classify_api_error(e)
             assert result.reason == FailoverReason.overloaded
     
    +    # ── 5xx that are actually request-validation errors ──
    +    # Some OpenAI-compatible gateways (e.g. codex.nekos.me) return
    +    # request-validation failures with a 5xx status. These are
    +    # deterministic, so they must NOT be retried — otherwise the retry
    +    # loop hammers the identical bad request into a flood.
    +
    +    def test_502_with_unknown_parameter_is_non_retryable(self):
    +        e = MockAPIError(
    +            "Unknown parameter: 'input[617]._empty_recovery_synthetic'",
    +            status_code=502,
    +            body={
    +                "error": {
    +                    "type": "invalid_request_error",
    +                    "message": (
    +                        "[ObjectParam] [input[617]._empty_recovery_synthetic] "
    +                        "[unknown_parameter] Unknown parameter: "
    +                        "'input[617]._empty_recovery_synthetic'."
    +                    ),
    +                }
    +            },
    +        )
    +        result = classify_api_error(e)
    +        assert result.reason == FailoverReason.format_error
    +        assert result.retryable is False
    +        assert result.should_fallback is True
    +
    +    def test_502_with_unsupported_parameter_is_non_retryable(self):
    +        e = MockAPIError(
    +            "Unsupported parameter: logprobs",
    +            status_code=502,
    +            body={
    +                "error": {
    +                    "type": "invalid_request_error",
    +                    "message": "Unsupported parameter: logprobs",
    +                }
    +            },
    +        )
    +        result = classify_api_error(e)
    +        assert result.reason == FailoverReason.format_error
    +        assert result.retryable is False
    +
    +    def test_500_with_invalid_request_error_type_is_non_retryable(self):
    +        e = MockAPIError(
    +            "bad request",
    +            status_code=500,
    +            body={"error": {"type": "invalid_request_error", "message": "bad request"}},
    +        )
    +        result = classify_api_error(e)
    +        assert result.reason == FailoverReason.format_error
    +        assert result.retryable is False
    +
    +    def test_502_plain_bad_gateway_still_retryable(self):
    +        """A genuine 502 with no request-validation signal stays retryable."""
    +        e = MockAPIError("Bad Gateway", status_code=502)
    +        result = classify_api_error(e)
    +        assert result.reason == FailoverReason.server_error
    +        assert result.retryable is True
    +
         # ── Model not found ──
     
         def test_404_model_not_found(self):
    ```
  - `tests/agent/transports/test_chat_completions.py`

    ```diff
    @@ -66,6 +66,38 @@ def test_convert_messages_strips_tool_name(self, transport):
             # Original list untouched (deepcopy-on-demand)
             assert msgs[2]["tool_name"] == "execute_code"
     
    +    def test_convert_messages_strips_internal_scaffolding_markers(self, transport):
    +        """Hermes-internal ``_``-prefixed markers must never reach the wire.
    +
    +        The empty-response recovery path appends synthetic messages tagged
    +        with ``_empty_recovery_synthetic``; permissive providers ignore the
    +        unknown key, but strict gateways (opencode-go, codex.nekos.me)
    +        reject the request, poisoning every later turn in the session.
    +        """
    +        msgs = [
    +            {"role": "user", "content": "run the task"},
    +            {"role": "assistant", "content": "(empty)", "_empty_recovery_synthetic": True},
    +            {"role": "user", "content": "continue", "_empty_recovery_synthetic": True},
    +            {"role": "assistant", "content": "done", "_thinking_prefill": True,
    +             "_empty_terminal_sentinel": True},
    +        ]
    +        result = transport.convert_messages(msgs)
    +        for m in result:
    +            assert not any(k.startswith("_") for k in m), m
    +        # Visible content preserved
    +        assert result[1]["content"] == "(empty)"
    +        assert result[2]["content"] == "continue"
    +        # Original list untouched (deepcopy-on-demand)
    +        assert msgs[1]["_empty_recovery_synthetic"] is True
    +
    +    def test_convert_messages_clean_list_is_identity(self, transport):
    +        """A list with no internal/codex keys is returned as-is (no copy)."""
    +        msgs = [
    +            {"role": "user", "content": "hi"},
    +            {"role": "assistant", "content": "hello"},
    +        ]
    +        assert transport.convert_messages(msgs) is msgs
    +
     
     class TestChatCompletionsBuildKwargs:
     
    ```
- **#48295: fix(transports/chat_completions): strip session timestamp from API-facing messages**. closed without merge · fix tier `suggested`
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#47875`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#48358: fix(transport): strip internal timestamp from API messages (#48263)**. closed without merge · fix tier `suggested`
  - maintainer direction: category `merged_via`; implication `superseded`; refs `#47875`, `#47875`, `#48295`, `#48331`, `#48263`, `#47868`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#49573: fix(transport): strip timestamp and observed from chat completions messages**. closed without merge · fix tier `suggested`
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#49595`, `#49572`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#49840: fix(chat-completions): strip internal timestamp field before strict providers**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `agent/transports/chat_completions.py`, `scripts/release.py`, `tests/agent/transports/test_chat_completions.py`
  - fix hunks:
  - `agent/transports/chat_completions.py`

    ```diff
    @@ -172,6 +172,7 @@ def convert_messages(
                     "codex_reasoning_items" in msg
                     or "codex_message_items" in msg
                     or "tool_name" in msg
    +                or "timestamp" in msg  # #47868 — strict providers reject this
                 ):
                     needs_sanitize = True
                     break
    ```

    ```diff
    @@ -201,6 +202,7 @@ def convert_messages(
                 msg.pop("codex_reasoning_items", None)
                 msg.pop("codex_message_items", None)
                 msg.pop("tool_name", None)
    +            msg.pop("timestamp", None)  # #47868 — leak into strict providers
                 # Drop all Hermes-internal scaffolding markers (``_``-prefixed).
                 # OpenAI's message schema has no ``_``-prefixed fields, so this
                 # is safe and future-proofs against new markers being added.
    ```
  - `scripts/release.py`

    ```diff
    @@ -46,6 +46,7 @@
     # Auto-extracted from noreply emails + manual overrides
     AUTHOR_MAP = {
         "[redacted:email]": "w31rdm4ch1nZ",
    +    "[redacted:email]": "x7peeps",
         "[redacted:email]": "rratmansky",
         "[redacted:email]": "lkz-de",
         "[redacted:email]": "salesondemandio",
    ```
  - `tests/agent/transports/test_chat_completions.py`

    ```diff
    @@ -104,6 +104,31 @@ def test_convert_messages_strips_tool_name(self, transport):
             # Original list untouched (deepcopy-on-demand)
             assert msgs[2]["tool_name"] == "execute_code"
     
    +    def test_convert_messages_strips_timestamp(self, transport):
    +        """Internal per-message ``timestamp`` metadata (stamped by
    +        ``_apply_persist_user_message_override`` to preserve platform event
    +        time without embedding it in content, and persisted to the SQLite
    +        store) is not part of the OpenAI Chat Completions schema. Strict
    +        providers like Mistral / Fireworks-backed endpoints reject it with
    +        HTTP 422 'Extra inputs are not permitted, field: messages[N].timestamp'.
    +        Regression test for #47868.
    +        """
    +        msgs = [
    +            {"role": "user", "content": "hi", "timestamp": 1781976577.0},
    +        ]
    +        result = transport.convert_messages(msgs)
    +        assert "timestamp" not in result[0]
    +        assert result[0]["content"] == "hi"
    +        assert result[0]["role"] == "user"
    +        # Original list untouched (deepcopy-on-demand)
    +        assert msgs[0]["timestamp"] == 1781976577.0
    +
    +    def test_convert_messages_no_copy_without_timestamp(self, transport):
    +        """A timestamp-free message list needs no sanitize pass and is
    +        returned by identity (preserves the deepcopy-on-demand contract)."""
    +        msgs = [{"role": "user", "content": "hi"}]
    +        assert transport.convert_messages(msgs) is msgs
    +
         def test_convert_messages_strips_internal_scaffolding_markers(self, transport):
             """Hermes-internal ``_``-prefixed markers must never reach the wire.
     
    ```
- **#55987: fix(agent): strip timestamp field from API messages (#55902)**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `4467c22c8`
- **#56112: fix(agent): strip timestamp field in summary path sanitizer**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `6d3d9d0baf18291e5306691b35f81e5b2e0eecb3`
- **#30982: fix(transport): strip internal bookkeeping fields before API calls**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `775a17284f00a0ab7014d0cea35a74f0c66cf3e5`, `#30959`, `#31456`
- **#47875: fix(chat-completions): strip timestamp from messages before sending to strict providers**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `4467c22c8f`, `#49840`, `#47868`
- **#47909: fix: strip internal message metadata before LLM calls**. closed without merge
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#47875`, `#47868`
- **#48170: fix(agent): strip timestamp metadata from chat payloads**. open
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#47875`, `#47868`, `#47868`, `#47875`
  - files: `agent/chat_completion_helpers.py`, `tests/run_agent/test_run_agent.py`
- **#48266: fix:sanitize internal fields (timestamp) from API messages #48263**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#47868`, `#48263`
- **#48592: fix(chat_completions): strip timestamp/observed/platform_message_id from API messages**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#47875`, `#47875`, `#48358`, `#47875`, `#48170`, `#47875`, `#47868`
- **#48604: fix(agent): strip gateway replay metadata from messages before API calls (#48523)**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#47875`, `#47875`, `#48592`, `#48358`, `#48575`, `#47875`, `#48523`, `#47868`
- **#48916: fix(prompt): honor explicit system prompt timestamp config**. open
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#48772`, `#48772`
  - files: `agent/system_prompt.py`, `cli-config.yaml.example`, `hermes_cli/config.py`, `tests/run_agent/test_run_agent.py`
- **#49595: fix(transport): strip timestamp and observed from chat completions messages**. open
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#47875`, `#47875`, `#47875`, `#47868`, `#49573`
  - files: `agent/transports/chat_completions.py`
- **#55907: fix(agent): strip message timestamp before API calls**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`
- **#68166: fix(agent): strip gateway-only timestamp/message_id keys before LLM call**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `4467c22c8f097cce5b670e81852d69bfbb6aadea`, `#68077`

## Fix status

- Merged to main in PR #31456 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-06-21`.
- Subsystem: `agent`