# NousResearch/hermes-agent · Gateway stale lock detection broken on macOS/Windows

## Open now

- 1 issue, 0 PRs open on GitHub.
- #31890: macOS gateway scoped locks can block adapters after PID reuse when start_time is null

These issues and fixes center on the gateway's stale lock detection failing on non-Linux platforms because process start-time retrieval relied on Linux-only mechanisms, causing PID reuse to go undetected and leaving stale locks in place. The work involves adding cross-platform fallbacks (psutil, ps, cmdline inspection) to correctly identify and evict stale scoped and platform locks.

## TL;DR

- 25 member(s): 1 open issue(s), 7 closed issue(s), 0 open PR(s), 2 merged PR(s), 15 PR(s) closed without merging.
- A fix was merged to main in PR #68650. Verify it addresses your case.
- **Recommended:** read PR #68650 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-24500`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text.
- Members outside that structural core may belong to a different underlying problem.
- Pairwise link density `0.10`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Open related issues

- #31890: macOS gateway scoped locks can block adapters after PID reuse when start_time is null

## Closed / duplicate issues

- #16376: \[Bug\]: macOS stale Telegram token lock can block gateway after PID reuse
- #16586: Stale gateway lock files never cleared on macOS — process-start-time check is Linux-only
- #18778: gateway: scoped lock PID-reuse guard is a no-op on macOS/Windows — stale lockfiles permanently block startup
- #21596: Stale platform-lock detection broken on macOS: PID reuse causes permanent startup failure
- #21613: macOS: Telegram gateway lock always has start_time=null, causing PID reuse false positives
- #24067: \[Bug\]: macOS PID lock check fails — system process occupies same PID, causing zombie lock on gateway restart
- #53763: Bug: Stale lock cleanup fails on macOS due to PID recycling

## Prior & competing PRs

- **#16423: fix(gateway): detect stale scoped locks via cmdline when start_time is absent**. closed without merge · fix tier `suggested`
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `653d30429039d1f5f048889e5397c1297c1fda38`, `#23953`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#16655: fix(gateway): detect stale lock files on macOS via \`ps -o lstart=\`**. closed without merge · fix tier `suggested`
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `#15144`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#24500: fix(gateway): detect stale scoped locks via cmdline when start_time is absent on macOS**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`
  - files: `gateway/status.py`, `scripts/release.py`, `tests/gateway/test_status.py`
  - fix hunks:
  - `gateway/status.py`

    ```diff
    @@ -124,16 +124,33 @@ def get_process_start_time(pid: int) -> Optional[int]:
     
     
     def _read_process_cmdline(pid: int) -> Optional[str]:
    -    """Return the process command line as a space-separated string."""
    +    """Return the process command line as a space-separated string.
    +
    +    On Linux, reads /proc/<pid>/cmdline directly.  On macOS and other
    +    platforms without /proc, falls back to ``ps -p <pid> -o command=``.
    +    """
         cmdline_path = Path(f"/proc/{pid}/cmdline")
         try:
             raw = cmdline_path.read_bytes()
         except (FileNotFoundError, PermissionError, OSError):
    -        return None
    +        pass
    +    else:
    +        if raw:
    +            return raw.replace(b"\x00", b" ").decode("utf-8", errors="ignore").strip()
     
    -    if not raw:
    -        return None
    -    return raw.replace(b"\x00", b" ").decode("utf-8", errors="ignore").strip()
    +    try:
    +        result = subprocess.run(
    +            ["ps", "-p", str(pid), "-o", "command="],
    +            capture_output=True,
    +            text=True,
    +            timeout=5,
    +        )
    +        if result.returncode == 0 and result.stdout.strip():
    +            return result.stdout.strip()
    +    except (OSError, subprocess.TimeoutExpired):
    +        pass
    +
    +    return None
     
     
     def _looks_like_gateway_process(pid: int) -> bool:
    ```

    ```diff
    @@ -594,6 +611,17 @@ def acquire_scoped_lock(scope: str, identity: str, metadata: Optional[dict[str,
                         and current_start != existing.get("start_time")
                     ):
                         stale = True
    +                # When start_time comparison is unavailable (macOS / Windows
    +                # have no /proc, so both sides are None), fall back to
    +                # checking the live process command line.  If the PID was
    +                # reused by an unrelated process the lock is stale.
    +                if (
    +                    not stale
    +                    and existing.get("start_time") is None
    +                    and current_start is None
    +                    and not _looks_like_gateway_process(existing_pid)
    +                ):
    +                    stale = True
                     # Check if process is stopped (Ctrl+Z / SIGTSTP) — stopped
                     # processes still appear alive to _pid_exists but are not
                     # actually running. Treat them as stale so --replace works.
    ```
  - `scripts/release.py`

    ```diff
    @@ -58,6 +58,7 @@
         "[redacted:email]": "hrkzogw",
         "[redacted:email]": "dandacompany",
         "[redacted:email]": "TreyDong",
    +    "[redacted:email]": "pkyanam",
         "127238744+teknium1@users.noreply.github.com": "teknium1",
         "[redacted:email]": "Hugo-SEQUIER",
         "128259593+Gutslabs@users.noreply.github.com": "Gutslabs",
    ```
  - `tests/gateway/test_status.py`

    ```diff
    @@ -444,6 +444,59 @@ def test_acquire_scoped_lock_rejects_live_other_process(self, tmp_path, monkeypa
             assert acquired is False
             assert existing["pid"] == 99999
     
    +    def test_acquire_scoped_lock_replaces_pid_reused_by_unrelated_process(self, tmp_path, monkeypatch):
    +        """macOS regression: PID reused by an unrelated process with start_time=None.
    +
    +        On macOS /proc is unavailable, so both the lock record and the live
    +        process report start_time=None.  The live PID is alive (os.kill
    +        succeeds) but belongs to a completely different program.  The lock
    +        must be treated as stale.
    +        """
    +        monkeypatch.setenv("HERMES_GATEWAY_LOCK_DIR", str(tmp_path / "locks"))
    +        lock_path = tmp_path / "locks" / "telegram-bot-token-2bb80d537b1da3e3.lock"
    +        lock_path.parent.mkdir(parents=True, exist_ok=True)
    +        lock_path.write_text(json.dumps({
    +            "pid": 873,
    +            "start_time": None,
    +            "kind": "hermes-gateway",
    +            "argv": ["/Users/user/.hermes/hermes-agent/hermes_cli/main.py", "gateway", "run", "--replace"],
    +        }))
    +
    +        # Post-#21561 the liveness probe routes through
    +        # ``gateway.status._pid_exists`` (psutil-first, safe on Windows),
    +        # not ``os.kill``.
    +        monkeypatch.setattr(status, "_pid_exists", lambda pid: True)
    +        monkeypatch.setattr(status, "_get_process_start_time", lambda pid: None)
    +        monkeypatch.setattr(status, "_looks_like_gateway_process", lambda pid: False)
    +
    +        acquired, existing = status.acquire_scoped_lock("telegram-bot-token", "secret", metadata={"platform": "telegram"})
    +
    +        assert acquired is True
    +        payload = json.loads(lock_path.read_text())
    +        assert payload["pid"] == os.getpid()
    +        assert payload["metadata"]["platform"] == "telegram"
    +
    +    def test_acquire_scoped_lock_keeps_lock_when_pid_reused_by_gateway(self, tmp_path, monkeypatch):
    +        """When start_time is None but the live PID still looks like a gateway, keep the lock."""
    +        monkeypatch.setenv("HERMES_GATEWAY_LOCK_DIR", str(tmp_path / "locks"))
    +        lock_path = tmp_path / "locks" / "telegram-bot-token-2bb80d537b1da3e3.lock"
    +        lock_path.parent.mkdir(parents=True, exist_ok=True)
    +        lock_path.write_text(json.dumps({
    +            "pid": 99999,
    +            "start_time": None,
    +            "kind": "hermes-gateway",
    +            "argv": ["/Users/user/.hermes/hermes-agent/hermes_cli/main.py", "gateway", "run", "--replace"],
    +        }))
    +
    +        monkeypatch.setattr(status, "_pid_exists", lambda pid: True)
    +        monkeypatch.setattr(status, "_get_process_start_time", lambda pid: None)
    +        monkeypatch.setattr(status, "_looks_like_gateway_process", lambda pid: True)
    +
    +        acquired, existing = status.acquire_scoped_lock("telegram-bot-token", "secret", metadata={"platform": "telegram"})
    +
    +        assert acquired is False
    +        assert existing["pid"] == 99999
    +
         def test_acquire_scoped_lock_replaces_stale_record(self, tmp_path, monkeypatch):
             monkeypatch.setenv("HERMES_GATEWAY_LOCK_DIR", str(tmp_path / "locks"))
             lock_path = tmp_path / "locks" / "telegram-bot-token-2bb80d537b1da3e3.lock"
    ```

    ```diff
    @@ -811,3 +864,46 @@ def raise_oserror(*args, **kwargs):
             ok = status.write_planned_stop_marker(target_pid=12345)
     
             assert ok is False
    +
    +
    +class TestReadProcessCmdlinePsFallback:
    +    """Tests for _read_process_cmdline falling back to ps on non-Linux."""
    +
    +    def test_ps_fallback_when_proc_unavailable(self, monkeypatch):
    +        monkeypatch.setattr(status.Path, "read_bytes", lambda self: (_ for _ in ()).throw(FileNotFoundError))
    +        monkeypatch.setattr(
    +            status.subprocess, "run",
    +            lambda args, **kwargs: SimpleNamespace(returncode=0, stdout="/usr/libexec/bluetoothuserd\n"),
    +        )
    +        result = status._read_process_cmdline(873)
    +        assert result == "/usr/libexec/bluetoothuserd"
    +
    +    def test_ps_fallback_returns_none_on_failure(self, monkeypatch):
    +        monkeypatch.setattr(status.Path, "read_bytes", lambda self: (_ for _ in ()).throw(FileNotFoundError))
    +        monkeypatch.setattr(
    +            status.subprocess, "run",
    +            lambda args, **kwargs: SimpleNamespace(returncode=1, stdout=""),
    +        )
    +        result = status._read_process_cmdline(99999)
    +        assert result is None
    +
    +    def test_proc_cmdline_takes_priority_over_ps(self, monkeypatch):
    +        calls = []
    +
    +        def fake_read_bytes(self):
    +            calls.append("proc")
    +            return b"python\x00hermes_cli/main.py\x00gateway\x00"
    +
    +        monkeypatch.setattr(status.Path, "read_bytes", fake_read_bytes)
    +        result = status._read_process_cmdline(12345)
    +        assert "hermes_cli/main.py" in result
    +        assert calls == ["proc"]
    +
    +    def test_ps_fallback_used_when_proc_returns_empty(self, monkeypatch):
    +        monkeypatch.setattr(status.Path, "read_bytes", lambda self: b"")
    +        monkeypatch.setattr(
    +            status.subprocess, "run",
    +            lambda args, **kwargs: SimpleNamespace(returncode=0, stdout="python hermes_cli/main.py gateway run\n"),
    +        )
    +        result = status._read_process_cmdline(12345)
    +        assert "hermes_cli/main.py" in result
    ```
- **#31923: fix(gateway): detect scoped-lock PID reuse on macOS when start_time is null (#31890)**. closed without merge · fix tier `suggested`
  - maintainer direction: category `superseded_by_pr`; implication `superseded`; refs `012f40c98`, `#012`, `#31890`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#68650: fix(gateway): stale scoped-lock detection for mixed-None start times + guarded lock reads + atomic removal**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`
  - files: `gateway/status.py`, `tests/gateway/test_status.py`
- **#16432: fix(gateway): detect stale scoped lock when PID is reused on macOS (#16376)**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#16423`, `#15144`, `#15584`
- **#16942: fix(gateway): add macOS/BSD fallback for process start-time and cmdline detection in lock files**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#16432`, `#12448`, `#15144`, `#16423`
- **#16966: fix(scoped_lock): treat null start_time lock as stale when live cmdline lacks hermes**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#16432`, `#16376`, `#16423`, `#15144`
- **#21018: fix(gateway): detect recycled macOS lock PIDs**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `653d30429039d1f5f048889e5397c1297c1fda38`, `#16586`
- **#22490: fix(gateway): use psutil for cross-platform process start time**. closed without merge
  - maintainer direction: category `uncategorized`; implication `needs_llm`
- **#22564: fix(gateway): macOS fallback for _get_process_start_time**. closed without merge
  - maintainer direction: category `duplicate_of`; implication `duplicate`; refs `#22490`, `#22564`
- **#23953: fix(gateway): detect stale scoped locks via cmdline when start_time is absent on macOS**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `653d30429`, `#24500`, `#21561`
- **#24595: fix(gateway): use psutil for cross-platform process start_time + cmdline detection**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`
- **#35625: fix(gateway): cross-platform process start-time for scoped-lock PID-reuse guard**. closed without merge
  - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `012f40c98c18b6723e355abbba7544b752836276`
- **#53767: fix(gateway): detect stale lock when macOS psutil returns valid start_time for recycled PID**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `#68650`, `#42689`
- **#54279: fix(gateway): treat null-start scoped locks as stale**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `#68650`, `#53767`
- **#56919: fix(gateway): honor cmdline fallback for null-start_time locks when live start_time resolves**. closed without merge
  - maintainer direction: category `merged_via`; implication `superseded`; refs `#68650`, `#53767`

## Fix status

- Merged to main in PR #68650 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-07-21`.
- Subsystem: `gateway`