# NousResearch/hermes-agent · Mid-session model switch persistence and token tracking ## Open now - Nothing open on GitHub. These issues and fixes address bugs where switching models mid-session fails to update billing provider, base URL, and model fields in the database, causing stale dashboard data. Related work adds per-model token usage tracking across mid-session model switches. ## TL;DR - 14 member(s): 0 open issue(s), 4 closed issue(s), 0 open PR(s), 3 merged PR(s), 7 PR(s) closed without merging. - A fix was merged to main in PR #35256. Verify it addresses your case. - **Recommended:** read PR #35256 (and any competing PRs below) before filing. Confirm the merged change covers your symptom. ## Maintainer resolution verdict - Internal-tracker verdict over this cluster's issue members: `resolved` · deciding PR(s): `pr-35256,pr-52666,pr-62610`. This is distinct from the fix status above. Weigh it yourself. ## How this cluster was grouped - Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text. - Members outside that structural core may belong to a different underlying problem. - Pairwise link density `0.13`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked. ## Closed / duplicate issues - #28637: Per-model token usage is lost during mid-session model switches - #34850: Bug: Model switch mid-session not persisted to database (dashboard shows stale model) - #48248: \[Bug\]: \`billing_provider\` / \`billing_base_url\` are write-once via COALESCE — mid-session \`/model\` switch leaves stale provider in session DB and misleads dashboard model cards - #51607: Session model/billing fields don't reflect mid-session model switches → incorrect metrics ## Prior & competing PRs - **#35256: fix(state): persist mid-session model switch to database (#34850)**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `gateway/run.py`, `hermes_state.py`, `scripts/release.py`, `tests/test_hermes_state.py` - fix hunks: - `gateway/run.py` ```diff @@ -10549,6 +10549,22 @@ async def _on_model_selected( except Exception as exc: logger.warning("Picker model switch failed for cached agent: %s", exc) + # Persist the new model to the session DB so the + # dashboard shows the updated model (#34850). + _sess_db = getattr(_self, "_session_db", None) + if _sess_db is not None: + try: + _sess_entry = _self.session_store.get_or_create_session( + event.source + ) + _sess_db.update_session_model( + _sess_entry.session_id, result.new_model + ) + except Exception as exc: + logger.debug( + "Failed to persist model switch to DB: %s", exc + ) + # Store model note + session override if not hasattr(_self, "_pending_model_notes"): _self._pending_model_notes = {} ``` ```diff @@ -10686,6 +10702,20 @@ async def _on_model_selected( except Exception as exc: logger.warning("In-place model switch failed for cached agent: %s", exc) + # Persist the new model to the session DB so the dashboard + # shows the updated model (#34850). + _sess_db = getattr(self, "_session_db", None) + if _sess_db is not None: + try: + _sess_entry = self.session_store.get_or_create_session(source) + _sess_db.update_session_model( + _sess_entry.session_id, result.new_model + ) + except Exception as exc: + logger.debug( + "Failed to persist model switch to DB: %s", exc + ) + # Store a note to prepend to the next user message so the model # knows about the switch (avoids system messages mid-history). if not hasattr(self, "_pending_model_notes"): ``` - `hermes_state.py` ```diff @@ -965,6 +965,20 @@ def _do(conn): ) self._execute_write(_do) + def update_session_model(self, session_id: str, model: str) -> None: + """Update the model for a session after a mid-session switch. + + Unlike ``update_token_counts`` which uses ``COALESCE(model, ?)`` + (only filling in NULL), this unconditionally sets the model column + so that the dashboard reflects the user's latest /model choice. + """ + def _do(conn): + conn.execute( + "UPDATE sessions SET model = ? WHERE id = ?", + (model, session_id), + ) + self._execute_write(_do) + def update_token_counts( self, session_id: str, ``` - `scripts/release.py` ```diff @@ -46,6 +46,7 @@ # Auto-extracted from noreply emails + manual overrides AUTHOR_MAP = { "[redacted:email]": "SeaXen", + "[redacted:email]": "LengR", "[redacted:email]": "HashClawAI", "[redacted:email]": "TonyPepeBear", "[redacted:email]": "nielskaspers", ``` - `tests/test_hermes_state.py` ```diff @@ -152,6 +152,30 @@ def test_update_token_counts_preserves_existing_model(self, db): session = db.get_session("s1") assert session["model"] == "anthropic/claude-opus-4.6" + def test_update_session_model_overwrites_existing(self, db): + """A mid-session /model switch must overwrite the stored model. + + update_token_counts uses COALESCE(model, ?) (first-writer-wins), so + the dashboard kept showing the original model after a switch (#34850). + update_session_model sets the column unconditionally. + """ + db.create_session(session_id="s1", source="telegram", + model="xiaomi/mimo-v2.5-pro") + # Token updates never change the model once set. + db.update_token_counts("s1", input_tokens=10, output_tokens=5, + model="xiaomi/mimo-v2.5-pro") + assert db.get_session("s1")["model"] == "xiaomi/mimo-v2.5-pro" + + # Explicit switch overwrites it. + db.update_session_model("s1", "xiaomi/mimo-v2.5") + assert db.get_session("s1")["model"] == "xiaomi/mimo-v2.5" + + # And a subsequent token update does NOT revert it (COALESCE no-ops + # because the column is now non-NULL). + db.update_token_counts("s1", input_tokens=10, output_tokens=5, + model="xiaomi/mimo-v2.5-pro") + assert db.get_session("s1")["model"] == "xiaomi/mimo-v2.5" + def test_parent_session(self, db): db.create_session(session_id="parent", source="cli") db.create_session(session_id="child", source="cli", parent_session_id="parent") ``` - **#52666: fix(hermes_state): persist billing route after mid-session /model switch (#48248)**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `agent/agent_runtime_helpers.py`, `hermes_state.py`, `tests/test_hermes_state.py` - fix hunks: - `agent/agent_runtime_helpers.py` ```diff @@ -1697,6 +1697,27 @@ def switch_model(agent, new_model, new_provider, api_key='', base_url='', api_mo old_model, old_provider, new_model, new_provider, ) + # ── Persist billing route to session DB ── + # The agent's _session_db / session_id may not be set in all contexts + # (tests, bare agents without a session DB, etc.). This ensures the + # dashboard Model cards show the actual provider after a mid-session + # /model switch instead of the stale session-creation provider. + # See #48248 for the full bug description. + _session_db = getattr(agent, "_session_db", None) + _session_id = getattr(agent, "session_id", None) + if _session_db is not None and _session_id: + try: + _session_db.update_session_billing_route( + _session_id, + provider=agent.provider, + base_url=agent.base_url, + billing_mode=getattr(agent, "api_mode", None), + ) + except Exception: + logger.warning( + "Failed to persist billing route after model switch", + exc_info=True, + ) def invoke_tool(agent, function_name: str, function_args: dict, effective_task_id: str, ``` - `hermes_state.py` ```diff @@ -1584,6 +1584,36 @@ def _do(conn): ) self._execute_write(_do) + def update_session_billing_route( + self, + session_id: str, + *, + provider: str, + base_url: str, + billing_mode: Optional[str] = None, + ) -> None: + """Unconditionally update the billing provider/base_url for a session. + + Unlike ``update_token_counts`` which uses ``COALESCE(billing_provider, ?)`` + (only filling in NULL), this unconditionally sets the billing fields so + that the dashboard reflects the user's latest /model switch. + + Also nulls ``system_prompt`` so the cached snapshot (which embeds a + stale ``Model:`` / ``Provider:`` header) is rebuilt — matching the + behavior of ``update_session_model`` (see #48173, #48248). + """ + def _do(conn): + conn.execute( + """UPDATE sessions SET + billing_provider = ?, + billing_base_url = ?, + billing_mode = COALESCE(?, billing_mode), + system_prompt = NULL + WHERE id = ?""", + (provider, base_url, billing_mode, session_id), + ) + self._execute_write(_do) + def update_token_counts( self, session_id: str, ``` - `tests/test_hermes_state.py` ```diff @@ -210,6 +210,54 @@ def test_update_session_model_overwrites_existing(self, db): model="xiaomi/mimo-v2.5-pro") assert db.get_session("s1")["model"] == "xiaomi/mimo-v2.5" + def test_update_session_billing_route_overwrites_after_switch(self, db): + """A mid-session provider switch must overwrite the billing route. + + update_token_counts writes billing fields with + COALESCE(billing_provider, ?) (first-writer-wins), so after a + provider switch the dashboard kept attributing cost to the original + provider (#48248). update_session_billing_route sets them + unconditionally and nulls system_prompt so the next turn rebuilds + the Model:/Provider: header (#48173). + """ + db.create_session(session_id="s1", source="telegram") + # First token update seeds the billing route. + db.update_token_counts("s1", input_tokens=10, output_tokens=5, + billing_provider="openrouter", + billing_base_url="https://openrouter.ai/api/v1", + billing_mode="api_key") + sess = db.get_session("s1") + assert sess["billing_provider"] == "openrouter" + # A later token update never changes it (COALESCE first-writer-wins). + db.update_token_counts("s1", input_tokens=10, output_tokens=5, + billing_provider="ollama", + billing_base_url="http://localhost:11434/v1", + billing_mode="local") + assert db.get_session("s1")["billing_provider"] == "openrouter" + + # Seed a stale prompt snapshot, then switch the billing route. + db.update_system_prompt("s1", "Model: x/old\nProvider: openrouter") + assert db.get_session("s1")["system_prompt"] is not None + db.update_session_billing_route( + "s1", provider="ollama", + base_url="http://localhost:11434/v1", billing_mode="local", + ) + sess = db.get_session("s1") + assert sess["billing_provider"] == "ollama" + assert sess["billing_base_url"] == "http://localhost:11434/v1" + assert sess["billing_mode"] == "local" + assert sess["system_prompt"] is None, \ + "system_prompt must be nulled so the next turn rebuilds Model:/Provider:" + + # billing_mode defaults to COALESCE — omitting it preserves the value. + db.update_session_billing_route( + "s1", provider="openai", + base_url="https://api.openai.com/v1", + ) + sess = db.get_session("s1") + assert sess["billing_provider"] == "openai" + assert sess["billing_mode"] == "local" # preserved (COALESCE on None) + def test_parent_session(self, db): db.create_session(session_id="parent", source="cli") db.create_session(session_id="child", source="cli", parent_session_id="parent") ``` - **#62610: fix(insights): attribute usage across model switches**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `agent/codex_runtime.py`, `agent/insights.py`, `hermes_state.py`, `scripts/release.py`, `tests/agent/test_insights.py`, `tests/test_hermes_state.py` - fix hunks: - `agent/codex_runtime.py` ```diff @@ -120,6 +120,9 @@ def _record_codex_app_server_usage(agent, turn) -> dict[str, Any]: agent._session_db.update_token_counts( agent.session_id, model=agent.model, + billing_provider=agent.provider, + billing_base_url=agent.base_url, + billing_mode="subscription_included", api_call_count=1, ) except Exception as exc: ``` - `agent/insights.py` ```diff @@ -17,6 +17,7 @@ """ import json +import sqlite3 import time from collections import Counter, defaultdict from datetime import datetime ``` ```diff @@ -141,8 +142,8 @@ def generate(self, days: int = 30, source: str = None) -> Dict[str, Any]: } # Compute insights - overview = self._compute_overview(sessions, message_stats) - models = self._compute_model_breakdown(sessions) + models = self._compute_model_breakdown(sessions, cutoff, source) + overview = self._compute_overview(sessions, message_stats, models) platforms = self._compute_platform_breakdown(sessions) tools = self._compute_tool_breakdown(tool_usage) skills = self._compute_skill_breakdown(skill_usage) ``` ```diff @@ -172,7 +173,7 @@ def generate(self, days: int = 30, source: str = None) -> Dict[str, Any]: "message_count, tool_call_count, input_tokens, output_tokens, " "cache_read_tokens, cache_write_tokens, billing_provider, " "billing_base_url, billing_mode, estimated_cost_usd, " - "actual_cost_usd, cost_status, cost_source") + "actual_cost_usd, cost_status, cost_source, api_call_count") # Pre-computed query strings — f-string evaluated once at class definition, # not at runtime, so no user-controlled value can alter the query structure. ``` ```diff @@ -399,7 +400,12 @@ def _get_message_stats(self, cutoff: float, source: str = None) -> Dict: # Computation # ========================================================================= - def _compute_overview(self, sessions: List[Dict], message_stats: Dict) -> Dict: + def _compute_overview( + self, + sessions: List[Dict], + message_stats: Dict, + models: Optional[List[Dict]] = None, + ) -> Dict: """Compute high-level overview statistics.""" total_input = sum(s.get("input_tokens") or 0 for s in sessions) total_output = sum(s.get("output_tokens") or 0 for s in sessions) ``` ```diff @@ -431,6 +437,9 @@ def _compute_overview(self, sessions: List[Dict], message_stats: Dict) -> Dict: else: models_without_pricing.add(display) + if models: + total_cost = sum(float(m.get("cost") or 0.0) for m in models) + # Session duration stats (guard against negative durations from clock drift) durations = [] for s in sessions: ``` ```diff @@ -473,39 +482,189 @@ def _compute_overview(self, sessions: List[Dict], message_stats: Dict) -> Dict: "included_cost_sessions": included_cost_sessions, } - def _compute_model_breakdown(self, sessions: List[Dict]) -> List[Dict]: - """Break down usage by model.""" + _GET_MODEL_USAGE_WITH_SOURCE = ( + "SELECT u.session_id, u.model, u.billing_provider, u.billing_base_url," + " u.api_call_count, u.input_tokens, u.output_tokens," + " u.cache_read_tokens, u.cache_write_tokens, u.reasoning_tokens," + " u.estimated_cost_usd, u.actual_cost_usd, u.cost_status," + " u.cost_source, u.billing_mode" + " FROM session_model_usage u" + " JOIN sessions s ON s.id = u.session_id" + " WHERE s.started_at >= ? AND s.source = ?" + ) + _GET_MODEL_USAGE_ALL = ( + "SELECT u.session_id, u.model, u.billing_provider, u.billing_base_url," + " u.api_call_count, u.input_tokens, u.output_tokens," + " u.cache_read_tokens, u.cache_write_tokens, u.reasoning_tokens," + " u.estimated_cost_usd, u.actual_cost_usd, u.cost_status," + " u.cost_source, u.billing_mode" + " FROM session_model_usage u" + " JOIN sessions s ON s.id = u.session_id" + " WHERE s.started_at >= ?" + ) + + def _get_model_usage(self, cutoff: float, source: str = None) -> List[Dict]: + """Fetch per-model usage rows within the window (issue #51607). + + Returns an empty list when the table is missing (e.g. a DB opened by + older code that never created it) so the caller can fall back to the + per-session aggregate. + """ + try: + if source: + cursor = self._conn.execute( + self._GET_MODEL_USAGE_WITH_SOURCE, (cutoff, source) + ) + else: + cursor = self._conn.execute(self._GET_MODEL_USAGE_ALL, (cutoff,)) + return [dict(row) for row in cursor.fetchall()] + except sqlite3.OperationalError: + return [] + + def _compute_model_breakdown( + self, sessions: List[Dict], cutoff: float, source: str = None + ) -> List[Dict]: + """Break down token usage and cost by model. + + Tokens and cost are attributed per model from session_model_usage, so a + session that switched models mid-flight (via ``/model``) splits across + every model it used instead of dumping everything on the initial model + (issue #51607). Sessions without per-model rows — e.g. data written + before this table existed and not yet backfilled — fall back to their + single recorded (model, billing_provider) aggregate so nothing is lost. + + Tool calls aren't tied to a specific API invocation, so they stay + attributed to the session's recorded model. + """ model_data = defaultdict(lambda: { - "sessions": 0, "input_tokens": 0, "output_tokens": 0, + "sessions": set(), "input_tokens": 0, "output_tokens": 0, "cache_read_tokens": 0, "cache_write_tokens": 0, - "total_tokens": 0, "tool_calls": 0, "cost": 0.0, + "reasoning_tokens": 0, "total_tokens": 0, "api_calls": 0, + "tool_calls": 0, "cost": 0.0, "actual_cost": 0.0, }) - for s in sessions: - model = s.get("model") or "unknown" + def _accumulate(model, provider, base_url, session_id, inp, out, + cache_read, cache_write, reasoning, *, + stored_cost=None, actual_cost=None, cost_status=None): + model = model or "unknown" # Normalize: strip provider prefix for display display_model = model.split("/")[-1] if "/" in model else model - d = model_data[display_model] - d["sessions"] += 1 - inp = s.get("input_tokens") or 0 - out = s.get("output_tokens") or 0 - cache_read = s.get("cache_read_tokens") or 0 - cache_write = s.get("cache_write_tokens") or 0 + d: Dict[str, Any] = model_data[display_model] + d["sessions"].add(session_id) d["input_tokens"] += inp d["output_tokens"] += out d["cache_read_tokens"] += cache_read d["cache_write_tokens"] += cache_write + d["reasoning_tokens"] += reasoning d["total_tokens"] += inp + out + cache_read + cache_write - d["tool_calls"] += s.get("tool_call_count") or 0 - estimate, status = _estimate_cost(s) + if stored_cost is None: + estimate, status = _estimate_cost( + model, inp, out, + cache_read_tokens=[redacted:secret-assignment] cache_write_tokens=[redacted:secret-assignment] + provider=provider or None, base_url=base_url, + ) + else: + estimate = float(stored_cost or 0.0) + status = cost_status or "unknown" d["cost"] += estimate - d["has_pricing"] = has_known_pricing(model, s.get("billing_provider"), s.get("billing_base_url")) + d["actual_cost"] += float(actual_cost or 0.0) d["cost_status"] = status + if has_known_pricing(model, provider or None, base_url): + d["has_pricing"] = True + else: + d.setdefault("has_pricing", False) + return display_model + + usage_rows = self._get_model_usage(cutoff, source) + usage_totals = defaultdict(lambda: { + "input_tokens": 0, "output_tokens": 0, "cache_read_tokens": 0, + "cache_write_tokens": 0, "reasoning_tokens": 0, + "api_call_count": 0, "estimated_cost_usd": 0.0, + "actual_cost_usd": 0.0, + }) + for r in usage_rows: + totals: Dict[str, Any] = usage_totals[r["session_id"]] + for key in ( + "input_tokens", "output_tokens", "cache_read_tokens", + "cache_write_tokens", "reasoning_tokens", "api_call_count", + ): + totals[key] += r[key] or 0 + totals["estimated_cost_usd"] += r["estimated_cost_usd"] or 0.0 + totals["actual_cost_usd"] += r["actual_cost_usd"] or 0.0 + d = _accumulate( + r["model"], r["billing_provider"], r.get("billing_base_url"), + r["session_id"], r["input_tokens"] or 0, r["output_tokens"] or 0, + r["cache_read_tokens"] or 0, r["cache_write_tokens"] or 0, + r["reasoning_tokens"] or 0, + stored_cost=( + r["estimated_cost_usd"] + if r.get("cost_status") or r.get("cost_source") + else None + ), + actual_cost=r["actual_cost_usd"], + cost_status=r.get("cost_status"), + ) + model_data[d]["api_calls"] += r["api_call_count"] or 0 - result = [ - {"model": model, **data} - for model, data in model_data.items() - ] + # Reconcile against the aggregate row. This covers legacy sessions, + # interrupted migrations, and absolute cumulative updates without + # double-counting already-attributed route deltas. + for s in sessions: + totals = usage_totals[s["id"]] + inp = max(0, (s.get("input_tokens") or 0) - totals["input_tokens"]) + out = max(0, (s.get("output_tokens") or 0) - totals["output_tokens"]) + cache_read = max( + 0, (s.get("cache_read_tokens") or 0) - totals["cache_read_tokens"] + ) + cache_write = max( + 0, (s.get("cache_write_tokens") or 0) - totals["cache_write_tokens"] + ) + residual_cost = max( + 0.0, float(s.get("estimated_cost_usd") or 0.0) + - totals["estimated_cost_usd"], + ) + residual_actual = max( + 0.0, float(s.get("actual_cost_usd") or 0.0) + - totals["actual_cost_usd"], + ) + residual_calls = max( + 0, (s.get("api_call_count") or 0) - totals["api_call_count"] + ) + if not ( + inp or out or cache_read or cache_write or residual_cost + or residual_actual or residual_calls + ): + continue + d = _accumulate( + s.get("model"), s.get("billing_provider"), + s.get("billing_base_url"), s["id"], + inp, out, cache_read, cache_write, 0, + stored_cost=residual_cost, + actual_cost=residual_actual, + cost_status=s.get("cost_status"), + ) + residual_bucket: Dict[str, Any] = model_data[d] + residual_bucket["api_calls"] += residual_calls + + # Tool calls are attributed by the session's recorded model. + for s in sessions: + tool_calls = s.get("tool_call_count") or 0 + if not tool_calls: + continue + model = s.get("model") or "unknown" + display_model = model.split("/")[-1] if "/" in model else model + model_data[display_model]["tool_calls"] += tool_calls + + result = [] + for model, data in model_data.items(): + entry = {"model": model, **data} + entry["sessions"] = len(data["sessions"]) + # Models that surfaced only via tool-call attribution (no token + # rows) won't have these set by _accumulate — default them so the + # output shape is uniform for downstream/JSON consumers. + entry.setdefault("has_pricing", False) + entry.setdefault("cost_status", "unknown") + result.append(entry) # Sort by tokens first, fall back to session count when tokens are 0 result.sort(key=lambda x: (x["total_tokens"], x["sessions"]), reverse=True) return result ``` - `hermes_state.py` ```diff @@ -122,7 +122,7 @@ def _delete_delegate_children(conn, parent_ids: List[str]) -> List[str]: DEFAULT_DB_PATH = get_hermes_home() / "state.db" -SCHEMA_VERSION = 19 +SCHEMA_VERSION = 20 # Cap on user-controlled FTS5 query input before regex/sanitizer processing. # Search queries do not need to be arbitrarily large, and bounding them keeps ``` ```diff @@ -768,6 +768,27 @@ def repair_state_db_schema(db_path: Path, *, backup: bool = True) -> Dict[str, A compacted INTEGER NOT NULL DEFAULT 0 ); +CREATE TABLE IF NOT EXISTS session_model_usage ( + session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE, + model TEXT NOT NULL, + billing_provider TEXT NOT NULL DEFAULT '', + billing_base_url TEXT NOT NULL DEFAULT '', + billing_mode TEXT NOT NULL DEFAULT '', + api_call_count INTEGER NOT NULL DEFAULT 0, + input_tokens INTEGER NOT NULL DEFAULT 0, + output_tokens INTEGER NOT NULL DEFAULT 0, + cache_read_tokens INTEGER NOT NULL DEFAULT 0, + cache_write_tokens INTEGER NOT NULL DEFAULT 0, + reasoning_tokens INTEGER NOT NULL DEFAULT 0, + estimated_cost_usd REAL NOT NULL DEFAULT 0, + actual_cost_usd REAL NOT NULL DEFAULT 0, + cost_status TEXT, + cost_source TEXT, + first_seen REAL, + last_seen REAL, + PRIMARY KEY (session_id, model, billing_provider, billing_base_url, billing_mode) +); + CREATE TABLE IF NOT EXISTS state_meta ( key TEXT PRIMARY KEY, value TEXT ``` ```diff @@ -794,6 +815,8 @@ def repair_state_db_schema(db_path: Path, *, backup: bool = True) -> Dict[str, A CREATE INDEX IF NOT EXISTS idx_sessions_started ON sessions(started_at DESC); CREATE INDEX IF NOT EXISTS idx_messages_session ON messages(session_id, timestamp); CREATE INDEX IF NOT EXISTS idx_compression_locks_expires ON compression_locks(expires_at); +CREATE INDEX IF NOT EXISTS idx_session_model_usage_session ON session_model_usage(session_id); +CREATE INDEX IF NOT EXISTS idx_session_model_usage_model ON session_model_usage(model); """ # Indexes that reference columns added in later schema versions must be ``` ```diff @@ -1554,6 +1577,50 @@ def _init_schema(self): # means consumers fall back to sessions.json for those # rows until the gateway rewrites them. logger.debug("v18 gateway metadata backfill skipped: %s", exc) + if current_version < 20: + # v20: per-model usage attribution (issue #51607). Going + # forward update_token_counts() records each API call into + # session_model_usage keyed by the live model, but existing + # sessions only have their aggregate totals on the sessions + # row. Seed one usage row per historical session from those + # aggregates so insights reads uniformly from the new table. + # INSERT OR IGNORE keeps it idempotent: if newer code already + # wrote a (session_id, model, provider) row for a session, the + # PK conflict skips the stale aggregate rather than doubling it. + try: + cursor.execute( + """INSERT OR IGNORE INTO session_model_usage ( + session_id, model, billing_provider, + billing_base_url, billing_mode, + api_call_count, input_tokens, + output_tokens, cache_read_tokens, + cache_write_tokens, reasoning_tokens, + estimated_cost_usd, actual_cost_usd, + cost_status, cost_source, first_seen, last_seen + ) + SELECT id, COALESCE(model, 'unknown'), + COALESCE(billing_provider, ''), + COALESCE(billing_base_url, ''), + COALESCE(billing_mode, ''), + COALESCE(api_call_count, 0), + COALESCE(input_tokens, 0), + COALESCE(output_tokens, 0), + COALESCE(cache_read_tokens, 0), + COALESCE(cache_write_tokens, 0), + COALESCE(reasoning_tokens, 0), + COALESCE(estimated_cost_usd, 0), + COALESCE(actual_cost_usd, 0), + cost_status, cost_source, + started_at, COALESCE(ended_at, started_at) + FROM sessions + WHERE COALESCE(input_tokens, 0) + + COALESCE(output_tokens, 0) + + COALESCE(cache_read_tokens, 0) + + COALESCE(cache_write_tokens, 0) + + COALESCE(reasoning_tokens, 0) > 0""" + ) + except sqlite3.OperationalError: + pass if current_version < SCHEMA_VERSION and fts_migrations_complete: cursor.execute( "UPDATE schema_version SET version = ?", ``` ```diff @@ -2481,6 +2548,11 @@ def update_token_counts( model = COALESCE(model, ?), api_call_count = COALESCE(api_call_count, 0) + ? WHERE id = ?""" + has_accounted_usage = bool( + input_tokens or output_tokens or cache_read_tokens + or cache_write_tokens or reasoning_tokens or api_call_count + or estimated_cost_usd or actual_cost_usd + ) params = ( input_tokens, output_tokens, ``` ```diff @@ -2493,17 +2565,167 @@ def update_token_counts( cost_status, cost_source, pricing_version, - billing_provider, - billing_base_url, - billing_mode, - model, + billing_provider if has_accounted_usage else None, + billing_base_url if has_accounted_usage else None, + billing_mode if has_accounted_usage else None, + model if has_accounted_usage else None, api_call_count, session_id, ) + # Per-model usage attribution. ``update_token_counts`` is the single + # chokepoint every per-API-call delta flows through (CLI, gateway, cron, + # delegated runs — see conversation_loop / codex_runtime), and each call + # carries the model/provider *active at the time of that call*. The + # ``sessions`` row only keeps one (model, billing_provider) pair, so a + # mid-session ``/model`` switch otherwise attributes every token to the + # initial model (issue #51607). Recording the per-call delta into + # session_model_usage keyed by the live model preserves an accurate + # per-model breakdown regardless of how many times the user switches. + # + # Only the incremental path records here. Absolute cumulative updates + # cannot be split back into routes; Insights reconciles any positive + # residual against the aggregate session row instead. + record_model_usage = (not absolute) and ( + input_tokens or output_tokens or cache_read_tokens + or cache_write_tokens or reasoning_tokens or api_call_count + or estimated_cost_usd + ) + def _do(conn): + row = conn.execute( + "SELECT model, billing_provider, api_call_count FROM sessions WHERE id = ?", + (session_id,), + ).fetchone() + existing_model = row["model"] if row is not None else None + existing_provider = row["billing_provider"] if row is not None else None + existing_api_calls = int((row["api_call_count"] if row is not None else 0) or 0) + + # Session creation records the requested primary route before any API + # call. If it fails and fallback succeeds, the first accounted usage + # event is the first authoritative route. After that, preserve the + # legacy row: one row cannot represent mixed-provider usage. + first_accounted_route = ( + existing_api_calls == 0 + and has_accounted_usage + and bool(model) + and bool(billing_provider) + and (existing_model != model or existing_provider != billing_provider) + ) + if first_accounted_route: + conn.execute( + """UPDATE sessions + SET model = ?, billing_provider = ?, + billing_base_url = ?, billing_mode = ? + WHERE id = ?""", + (model, billing_provider, billing_base_url, billing_mode, session_id), + ) conn.execute(sql, params) + if record_model_usage: + self._record_model_usage( + conn, + session_id, + model=model, + billing_provider=billing_provider, + billing_base_url=billing_base_url, + billing_mode=billing_mode, + input_tokens=[redacted:secret-assignment] + output_tokens=[redacted:secret-assignment] + cache_read_tokens=[redacted:secret-assignment] + cache_write_tokens=[redacted:secret-assignment] + reasoning_tokens=[redacted:secret-assignment] + estimated_cost_usd=estimated_cost_usd, + actual_cost_usd=actual_cost_usd, + cost_status=cost_status, + cost_source=cost_source, + api_call_count=api_call_count, + ) self._execute_write(_do) + def _record_model_usage( + self, + conn, + session_id: str, + *, + model: Optional[str], + billing_provider: Optional[str], + billing_base_url: Optional[str], + billing_mode: Optional[str], + input_tokens: int, + output_tokens: int, + cache_read_tokens: int, + cache_write_tokens: int, + reasoning_tokens: int, + estimated_cost_usd: Optional[float], + actual_cost_usd: Optional[float], + cost_status: Optional[str], + cost_source: Optional[str], + api_call_count: int, + ) -> None: + """Accumulate a per-API-call usage delta into session_model_usage. + + Runs inside the caller's write transaction (after the ``sessions`` + UPDATE) so the per-model rows stay consistent with the summary row. + When the caller omits the model/provider (some paths only pass token + deltas), fall back to the values already recorded on the session row — + the same COALESCE-from-session behaviour the summary update uses. + """ + row = conn.execute( + "SELECT model, billing_provider, billing_base_url, billing_mode " + "FROM sessions WHERE id = ?", + (session_id,), + ).fetchone() + sess_model = row["model"] if row is not None else None + sess_provider = row["billing_provider"] if row is not None else None + sess_base_url = row["billing_base_url"] if row is not None else None + sess_billing_mode = row["billing_mode"] if row is not None else None + + eff_model = model or sess_model or "unknown" + eff_provider = billing_provider or sess_provider or "" + eff_base_url = billing_base_url or sess_base_url or "" + eff_billing_mode = billing_mode or sess_billing_mode or "" + now = time.time() + conn.execute( + """INSERT INTO session_model_usage ( + session_id, model, billing_provider, billing_base_url, billing_mode, + api_call_count, input_tokens, output_tokens, + cache_read_tokens, cache_write_tokens, reasoning_tokens, + estimated_cost_usd, actual_cost_usd, cost_status, cost_source, + first_seen, last_seen + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(session_id, model, billing_provider, billing_base_url, billing_mode) + DO UPDATE SET + api_call_count = api_call_count + excluded.api_call_count, + input_tokens = [redacted:secret-assignment] + excluded.input_tokens, + output_tokens = [redacted:secret-assignment] + excluded.output_tokens, + cache_read_tokens = [redacted:secret-assignment] + excluded.cache_read_tokens, + cache_write_tokens = [redacted:secret-assignment] + excluded.cache_write_tokens, + reasoning_tokens = [redacted:secret-assignment] + excluded.reasoning_tokens, + estimated_cost_usd = estimated_cost_usd + excluded.estimated_cost_usd, + actual_cost_usd = actual_cost_usd + excluded.actual_cost_usd, + cost_status = COALESCE(excluded.cost_status, cost_status), + cost_source = COALESCE(excluded.cost_source, cost_source), + last_seen = excluded.last_seen""", + ( + session_id, + eff_model, + eff_provider, + eff_base_url, + eff_billing_mode, + api_call_count or 0, + input_tokens or 0, + output_tokens or 0, + cache_read_tokens or 0, + cache_write_tokens or 0, + reasoning_tokens or 0, + float(estimated_cost_usd or 0.0), + float(actual_cost_usd or 0.0), + cost_status, + cost_source, + now, + now, + ), + ) + def ensure_session( self, session_id: str, ``` - `scripts/release.py` ```diff @@ -1078,6 +1078,8 @@ "[redacted:email]": "alexzhu0", "[redacted:email]": "AntAISecurityLab", "[redacted:email]": "briandevans", + "[redacted:email]": "tcconnally", + "[redacted:email]": "mijanx", "[redacted:email]": "inchargeautomation-lab", "[redacted:email]": "Bartok9", "[redacted:email]": "sweetcornna", ``` ```diff @@ -1509,6 +1511,8 @@ "[redacted:email]": "DaniuXie", "[redacted:email]": "ChanlerDev", "[redacted:email]": "briandevans", + "[redacted:email]": "tcconnally", + "[redacted:email]": "mijanx", "[redacted:email]": "h0tp-ftw", "[redacted:email]": "chinadbo", "[redacted:email]": "kshitijk4poor", ``` - `tests/agent/test_insights.py` ```diff @@ -320,6 +320,76 @@ def test_model_breakdown(self, populated_db): claude = next(m for m in models if "claude-sonnet" in m["model"]) assert claude["sessions"] == 2 + def test_model_breakdown_splits_mid_session_switch(self, db): + """A session that switches models mid-flight is split across both + models in the breakdown, not dumped on the initial model (#51607). + """ + now = time.time() + db.create_session(session_id="sw", source="cli", + model="deepseek/deepseek-v4-pro") + # 40k tokens on deepseek, then switch and 50k on opus. + db.update_token_counts("sw", input_tokens=[redacted:secret-assignment] output_tokens=8000, + model="deepseek/deepseek-v4-pro", + billing_provider="deepseek", api_call_count=2) + db.update_session_model("sw", "anthropic/claude-opus-4.8") + db.update_token_counts("sw", input_tokens=[redacted:secret-assignment] output_tokens=4000, + model="anthropic/claude-opus-4.8", + billing_provider="openrouter", api_call_count=3) + db._conn.commit() + + report = InsightsEngine(db).generate(days=30) + models = {m["model"]: m for m in report["models"]} + assert "deepseek-v4-pro" in models + assert "claude-opus-4.8" in models + # Tokens attributed to the model that actually incurred them. + assert models["deepseek-v4-pro"]["input_tokens"] == 40000 + assert models["claude-opus-4.8"]["input_tokens"] == 50000 + assert models["claude-opus-4.8"]["api_calls"] == 3 + # The summary row's single model would have hidden one of these. + assert models["deepseek-v4-pro"]["total_tokens"] == 48000 + assert models["claude-opus-4.8"]["total_tokens"] == 54000 + + def test_partial_per_model_rows_preserve_session_totals(self, db): + """A partial rolling-upgrade row must not hide aggregate residuals.""" + db.create_session(session_id="partial", source="cli", model="gpt-4o") + db.update_token_counts( + "partial", input_tokens=100, output_tokens=20, + model="gpt-4o", billing_provider="openai", api_call_count=1, + ) + db.update_token_counts( + "partial", input_tokens=1000, output_tokens=200, + model="gpt-4o", billing_provider="openai", api_call_count=10, + absolute=True, + ) + + report = InsightsEngine(db).generate(days=30) + model = next(m for m in report["models"] if m["model"] == "gpt-4o") + assert model["input_tokens"] == 1000 + assert model["output_tokens"] == 200 + assert model["api_calls"] == 10 + assert sum(m["total_tokens"] for m in report["models"]) == \ + report["overview"]["total_tokens"] + + def test_overview_cost_matches_per_model_stored_cost(self, db): + db.create_session(session_id="cost", source="cli", model="model-a") + db.update_token_counts( + "cost", input_tokens=10, model="model-a", billing_provider="custom", + estimated_cost_usd=1.25, actual_cost_usd=1.0, + cost_status="estimated", cost_source="provider", api_call_count=1, + ) + db.update_session_model("cost", "model-b") + db.update_session_billing_route("cost", provider="custom-b", base_url=None) + db.update_token_counts( + "cost", input_tokens=20, model="model-b", billing_provider="custom-b", + estimated_cost_usd=2.5, actual_cost_usd=2.0, + cost_status="estimated", cost_source="provider", api_call_count=1, + ) + + report = InsightsEngine(db).generate(days=30) + assert sum(m["cost"] for m in report["models"]) == pytest.approx(3.75) + assert report["overview"]["estimated_cost"] == pytest.approx(3.75) + assert report["overview"]["actual_cost"] == pytest.approx(3.0) + def test_platform_breakdown(self, populated_db): engine = InsightsEngine(populated_db) report = engine.generate(days=30) ``` - `tests/test_hermes_state.py` ```diff @@ -272,6 +272,43 @@ def test_update_token_counts_backfills_model_when_null(self, db): session = db.get_session("s1") assert session["model"] == "openai/gpt-5.4" + def test_first_accounted_fallback_replaces_requested_primary_route(self, db): + """First successful fallback usage must persist one coherent route pair.""" + db.create_session(session_id="s1", source="cli", model="gpt-5.6-sol") + + db.update_token_counts( + "s1", + input_tokens=10, + output_tokens=5, + model="glm-5.2", + billing_provider="custom:zai", + billing_base_url="https://api.z.ai/api/coding/paas/v4/", + api_call_count=1, + ) + + session = db.get_session("s1") + assert session["model"] == "glm-5.2" + assert session["billing_provider"] == "custom:zai" + assert session["billing_base_url"] == "https://api.z.ai/api/coding/paas/v4/" + assert session["api_call_count"] == 1 + + def test_accounted_primary_route_is_not_rewritten_by_later_fallback(self, db): + """A mixed-provider session keeps its first accounted route in the legacy row.""" + db.create_session(session_id="s1", source="cli", model="gpt-5.6-sol") + db.update_token_counts( + "s1", input_tokens=10, output_tokens=5, model="gpt-5.6-sol", + billing_provider="openai-codex", api_call_count=1, + ) + db.update_token_counts( + "s1", input_tokens=10, output_tokens=5, model="glm-5.2", + billing_provider="custom:zai", api_call_count=1, + ) + + session = db.get_session("s1") + assert session["model"] == "gpt-5.6-sol" + assert session["billing_provider"] == "openai-codex" + assert session["api_call_count"] == 2 + def test_update_token_counts_preserves_existing_model(self, db): db.create_session(session_id="s1", source="cli", model="anthropic/claude-opus-4.6") db.update_token_counts("s1", input_tokens=10, output_tokens=5, model="openai/gpt-5.4") ``` ```diff @@ -351,6 +388,181 @@ def test_update_session_billing_route_overwrites_after_switch(self, db): assert sess["billing_provider"] == "openai" assert sess["billing_mode"] == "local" # preserved (COALESCE on None) + def test_per_model_usage_recorded_for_single_model(self, db): + """Each per-call delta lands in session_model_usage (#51607).""" + db.create_session(session_id="s1", source="cli") + db.update_token_counts("s1", input_tokens=200, output_tokens=100, + model="anthropic/claude-opus-4.8", + billing_provider="anthropic", api_call_count=1) + db.update_token_counts("s1", input_tokens=100, output_tokens=50, + model="anthropic/claude-opus-4.8", + billing_provider="anthropic", api_call_count=1) + + rows = db._conn.execute( + "SELECT model, billing_provider, api_call_count, input_tokens, " + "output_tokens FROM session_model_usage WHERE session_id = 's1'" + ).fetchall() + assert len(rows) == 1 + row = rows[0] + assert row["model"] == "anthropic/claude-opus-4.8" + assert row["billing_provider"] == "anthropic" + assert row["api_call_count"] == 2 + assert row["input_tokens"] == 300 + assert row["output_tokens"] == 150 + + def test_mid_session_switch_splits_per_model_usage(self, db): + """The headline #51607 case: tokens after a /model switch are + attributed to the new model, not the session's initial model. + + The ``sessions`` summary row still holds combined totals + the latest + model, but session_model_usage keeps an accurate per-model split. + """ + db.create_session(session_id="s1", source="cli", + model="deepseek/deepseek-v4-pro") + # Pre-switch calls on deepseek. + db.update_token_counts("s1", input_tokens=[redacted:secret-assignment] output_tokens=[redacted:secret-assignment] + model="deepseek/deepseek-v4-pro", + billing_provider="deepseek", api_call_count=2) + # User runs /model — the gateway persists the new model … + db.update_session_model("s1", "anthropic/claude-opus-4.8") + # … and subsequent per-call deltas carry the new model/provider. + db.update_token_counts("s1", input_tokens=[redacted:secret-assignment] output_tokens=[redacted:secret-assignment] + model="anthropic/claude-opus-4.8", + billing_provider="openrouter", api_call_count=3) + + rows = { + r["model"]: r + for r in db._conn.execute( + "SELECT model, billing_provider, input_tokens, output_tokens, " + "api_call_count FROM session_model_usage WHERE session_id = 's1'" + ).fetchall() + } + assert set(rows) == {"deepseek/deepseek-v4-pro", + "anthropic/claude-opus-4.8"} + assert rows["deepseek/deepseek-v4-pro"]["input_tokens"] == 40_000 + assert rows["deepseek/deepseek-v4-pro"]["api_call_count"] == 2 + assert rows["anthropic/claude-opus-4.8"]["input_tokens"] == 50_000 + assert rows["anthropic/claude-opus-4.8"]["billing_provider"] == "openrouter" + assert rows["anthropic/claude-opus-4.8"]["api_call_count"] == 3 + + # Summary row: latest model + combined totals (unchanged behaviour). + session = db.get_session("s1") + assert session["model"] == "anthropic/claude-opus-4.8" + assert session["input_tokens"] == 90_000 + assert session["output_tokens"] == 12_000 + + def test_per_model_usage_falls_back_to_session_model(self, db): + """When a call omits the model, attribute it to the session's + recorded model — matches the COALESCE-from-session summary behaviour + and keeps existing callers (which pass no model) working. + """ + db.create_session(session_id="s1", source="cli", + model="gpt-4o", ) + db.update_token_counts("s1", input_tokens=10, output_tokens=5) + + rows = db._conn.execute( + "SELECT model FROM session_model_usage WHERE session_id = 's1'" + ).fetchall() + assert len(rows) == 1 + assert rows[0]["model"] == "gpt-4o" + + def test_absolute_update_does_not_record_per_model(self, db): + """absolute=True overwrites the cumulative summary row (gateway path) + and must NOT add per-model rows — those are accumulated from the + per-call incremental path, so recording here would double-count. + """ + db.create_session(session_id="s1", source="cli", model="gpt-4o") + db.update_token_counts("s1", input_tokens=500, output_tokens=200, + model="gpt-4o", absolute=True) + + rows = db._conn.execute( + "SELECT COUNT(*) AS n FROM session_model_usage WHERE session_id = 's1'" + ).fetchone() + assert rows["n"] == 0 + + def test_per_model_usage_keeps_distinct_billing_routes(self, db): + """The same model through distinct billing routes must not collapse.""" + db.create_session(session_id="routes", source="cli", model="shared-model") + db.update_token_counts( + "routes", input_tokens=10, model="shared-model", + billing_provider="custom", billing_base_url="https://one.example/v1", + billing_mode="api_key", estimated_cost_usd=0.01, api_call_count=1, + ) + db.update_token_counts( + "routes", input_tokens=20, model="shared-model", + billing_provider="custom", billing_base_url="https://two.example/v1", + billing_mode="subscription_included", estimated_cost_usd=0.0, + cost_status="included", api_call_count=1, + ) + + rows = db._conn.execute( + "SELECT billing_base_url, billing_mode, input_tokens " + "FROM session_model_usage WHERE session_id = 'routes' " + "ORDER BY billing_base_url" + ).fetchall() + assert [(r["billing_base_url"], r["billing_mode"], r["input_tokens"]) + for r in rows] == [ + ("https://one.example/v1", "api_key", 10), + ("https://two.example/v1", "subscription_included", 20), + ] + + def test_metadata_only_update_does_not_replace_requested_route(self, db): + db.create_session(session_id="metadata", source="cli", model="primary") + db.update_token_counts( + "metadata", model="fallback", billing_provider="fallback-provider", + api_call_count=0, + ) + row = db.get_session("metadata") + assert row["model"] == "primary" + assert row["billing_provider"] is None + + def test_first_accounted_route_replaces_all_route_fields_atomically(self, db): + db.create_session(session_id="route", source="cli", model="primary") + db.update_session_billing_route( + "route", provider="primary-provider", + base_url="https://primary.example/v1", billing_mode="api_key", + ) + db.update_token_counts( + "route", model="fallback", billing_provider="fallback-provider", + billing_base_url=None, billing_mode=None, api_call_count=1, + ) + row = db.get_session("route") + assert row["model"] == "fallback" + assert row["billing_provider"] == "fallback-provider" + assert row["billing_base_url"] is None + assert row["billing_mode"] is None + + def test_v17_backfill_seeds_existing_session_usage(self, tmp_path): + """A DB upgraded from <17 seeds one usage row per historical session + from its aggregate totals, so insights read uniformly from the table. + """ + db_path = tmp_path / "legacy.db" + db = SessionDB(db_path=db_path) + db.create_session(session_id="legacy1", source="cli", model="gpt-4o") + db.update_token_counts("legacy1", input_tokens=1234, output_tokens=567, + model="gpt-4o", billing_provider="openai") + # Simulate a pre-v17 database: drop the per-model rows and roll the + # recorded schema version back so the backfill migration re-runs. + db._conn.execute("DELETE FROM session_model_usage") + db._conn.execute("UPDATE schema_version SET version = 16") + db._conn.commit() + db.close() + + # Reopen — _init_schema should backfill from the sessions aggregate. + db2 = SessionDB(db_path=db_path) + try: + rows = db2._conn.execute( + "SELECT model, billing_provider, input_tokens, output_tokens " + "FROM session_model_usage WHERE session_id = 'legacy1'" + ).fetchall() + assert len(rows) == 1 + assert rows[0]["model"] == "gpt-4o" + assert rows[0]["billing_provider"] == "openai" + assert rows[0]["input_tokens"] == 1234 + assert rows[0]["output_tokens"] == 567 + finally: + db2.close() + def test_parent_session(self, db): db.create_session(session_id="parent", source="cli") db.create_session(session_id="child", source="cli", parent_session_id="parent") ``` ```diff @@ -1894,6 +2106,18 @@ def test_delete_session(self, db): assert db.get_session("s1") is None assert db.message_count(session_id="s1") == 0 + def test_delete_session_cascades_per_model_usage(self, db): + db.create_session(session_id="usage", source="cli", model="gpt-5") + db.update_token_counts( + "usage", input_tokens=10, model="gpt-5", + billing_provider="openai", api_call_count=1, + ) + assert db.delete_session("usage") is True + count = db._conn.execute( + "SELECT COUNT(*) FROM session_model_usage WHERE session_id = 'usage'" + ).fetchone()[0] + assert count == 0 + def test_delete_nonexistent(self, db): assert db.delete_session("nope") is False ``` - **#28842: Per-model token tracking for mid-session model switches**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#35256`, `#34850` - **#35181: fix(state): persist mid-session model switch to database**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `79f643067`, `#35256` - **#48254: fix(hermes_state): persist billing provider/base_url after mid-session /model switch**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `c7e934a5b`, `#52666` - **#48258: fix(state): remove COALESCE guard on billing fields for mid-session provider switch**. closed without merge - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#52666`, `#48248` - **#48357: fix(state): allow billing fields to update on model switch (#48248)**. closed without merge - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#48258`, `#48258`, `#48254`, `#48248`, `#48248` - **#49682: fix(state): use COALESCE(?, model) so mid-session model switches are persisted**. closed without merge - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `#34850`, `#35256` - **#51634: feat(agent): track per-model token usage for mid-session model switches**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `#62610` ## Fix status - Merged to main in PR #35256 (cited above). No release data is tracked. Verify downstream. - Fix last verified (merged) `2026-07-11`. - Subsystem: `agent`