# NousResearch/hermes-agent · Docker container cleanup on run failure

## Open now

- Nothing open on GitHub.

This cluster addresses a bug where Docker containers are left in a 'Created' (orphaned) state when 'docker run' or session initialization fails. Multiple fix attempts and follow-up PRs propose cleanup logic to handle these leaked containers.

## TL;DR

- 5 member(s): 0 open issue(s), 1 closed issue(s), 0 open PR(s), 1 merged PR(s), 3 PR(s) closed without merging.
- A fix was merged to main in PR #39412. Verify it addresses your case.
- **Recommended:** read PR #39412 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.

## Maintainer resolution verdict

- Internal-tracker verdict over this cluster's issue members: `resolved` · deciding PR(s): `pr-39412`. This is distinct from the fix status above. Weigh it yourself.

## How this cluster was grouped

- Link evidence (`STRUCTURAL_SPANNING`): every member is connected by structural evidence: shared changed files, or a closing/superseded reference.
- Pairwise link density `0.40`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.

## Closed / duplicate issues

- #7439: Docker containers leak in Created state when docker run fails

## Prior & competing PRs

- **#7440: fix(docker): clean up orphaned container when docker run fails**. closed without merge · fix tier `suggested`
  - maintainer direction: category `uncategorized`; implication `needs_llm`; refs `#7498`, `#7439`
  - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._
- **#39412: fix(docker): clean up orphaned container when docker run fails (salvage #7440)**. merged to main · fix tier `verified`
  - maintainer direction: category `merged`; implication `superseded`
  - files: `tests/tools/test_docker_environment.py`, `tools/environments/docker.py`
  - fix hunks:
  - `tests/tools/test_docker_environment.py`

    ```diff
    @@ -786,6 +786,84 @@ def test_reuse_falls_back_to_fresh_run_when_start_fails(monkeypatch):
         assert run_invocations, "fallback to fresh docker run must happen on start failure"
     
     
    +def test_failed_docker_run_cleans_up_orphaned_container(monkeypatch):
    +    """When ``docker run`` fails (e.g. exit 125), the partially-created
    +    container must be removed by name.
    +
    +    Docker can create the container object before failing to start it,
    +    leaving a stale ``Created`` container. The exited-only orphan reaper
    +    (``reap_orphan_containers``, ``status=exited``) never catches a
    +    ``Created`` orphan, so without this cleanup it leaks permanently.
    +    Regression for #7439. Salvage of #7440 (@Tranquil-Flow).
    +    """
    +    monkeypatch.setattr(docker_env, "find_docker", lambda: "/usr/bin/docker")
    +    monkeypatch.setattr(docker_env, "_get_active_profile_name", lambda: "default")
    +
    +    cleanup_calls = []
    +
    +    def _run(cmd, **kwargs):
    +        if isinstance(cmd, list) and len(cmd) >= 2:
    +            sub = cmd[1]
    +            if sub == "version":
    +                return subprocess.CompletedProcess(cmd, 0, stdout="Docker version", stderr="")
    +            if sub == "ps":
    +                # No reusable container -> fall through to a fresh `docker run`.
    +                return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
    +            if sub == "run":
    +                raise subprocess.CalledProcessError(
    +                    125, cmd, output="", stderr="docker: Error response from daemon"
    +                )
    +            if sub == "rm":
    +                cleanup_calls.append(list(cmd))
    +                return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
    +        return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
    +
    +    monkeypatch.setattr(docker_env.subprocess, "run", _run)
    +
    +    with pytest.raises(subprocess.CalledProcessError):
    +        _make_dummy_env()
    +
    +    assert len(cleanup_calls) == 1, "docker rm should be called once for the orphaned container"
    +    rm_cmd = cleanup_calls[0]
    +    assert rm_cmd[1] == "rm" and rm_cmd[2] == "-f"
    +    assert rm_cmd[3].startswith("hermes-"), "should remove the container by its generated name"
    +
    +
    +def test_docker_run_timeout_cleans_up_orphaned_container(monkeypatch):
    +    """When ``docker run`` times out (e.g. slow image pull), the
    +    partially-created container must be removed. Salvage of #7440
    +    (@Tranquil-Flow); regression for #7439.
    +    """
    +    monkeypatch.setattr(docker_env, "find_docker", lambda: "/usr/bin/docker")
    +    monkeypatch.setattr(docker_env, "_get_active_profile_name", lambda: "default")
    +
    +    cleanup_calls = []
    +
    +    def _run(cmd, **kwargs):
    +        if isinstance(cmd, list) and len(cmd) >= 2:
    +            sub = cmd[1]
    +            if sub == "version":
    +                return subprocess.CompletedProcess(cmd, 0, stdout="Docker version", stderr="")
    +            if sub == "ps":
    +                return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
    +            if sub == "run":
    +                raise subprocess.TimeoutExpired(cmd, 120)
    +            if sub == "rm":
    +                cleanup_calls.append(list(cmd))
    +                return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
    +        return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
    +
    +    monkeypatch.setattr(docker_env.subprocess, "run", _run)
    +
    +    with pytest.raises(subprocess.TimeoutExpired):
    +        _make_dummy_env()
    +
    +    assert len(cleanup_calls) == 1, "docker rm should be called once for the orphaned container"
    +    rm_cmd = cleanup_calls[0]
    +    assert rm_cmd[1] == "rm" and rm_cmd[2] == "-f"
    +    assert rm_cmd[3].startswith("hermes-"), "should remove the container by its generated name"
    +
    +
     def test_no_reuse_when_persist_across_processes_disabled(monkeypatch):
         """Opt-out path: ``persist_across_processes=False`` skips the ps probe
         entirely and always starts a fresh container, matching the pre-fix
    ```
  - `tools/environments/docker.py`

    ```diff
    @@ -854,13 +854,31 @@ def __init__(
                     "sleep", "infinity",  # no fixed lifetime — idle reaper handles cleanup
                 ]
                 logger.debug(f"Starting container: {' '.join(run_cmd)}")
    -            result = subprocess.run(
    -                run_cmd,
    -                capture_output=True,
    -                text=True,
    -                timeout=120,  # image pull may take a while
    -                check=True,
    -            )
    +            try:
    +                result = subprocess.run(
    +                    run_cmd,
    +                    capture_output=True,
    +                    text=True,
    +                    timeout=120,  # image pull may take a while
    +                    check=True,
    +                )
    +            except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
    +                # Docker may create the container object before `docker run`
    +                # fails to start it (e.g. exit code 125 when the daemon isn't
    +                # ready, or a timeout mid-pull). That orphan is left in
    +                # "Created" state — which the exited-only orphan reaper
    +                # (reap_orphan_containers, status=exited) never catches, so it
    +                # leaks permanently. Remove it by its known name before
    +                # re-raising. See #7439.
    +                logger.warning(
    +                    "docker run failed for %s, cleaning up orphaned container: %s",
    +                    container_name, e,
    +                )
    +                subprocess.run(
    +                    [self._docker_exe, "rm", "-f", container_name],
    +                    capture_output=True, timeout=10,
    +                )
    +                raise
                 self._container_id = result.stdout.strip()
                 logger.info(f"Started container {container_name} ({self._container_id[:12]})")
     
    ```
- **#7498: fix(docker): clean up containers when docker run or init_session fails**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`
- **#11535: fix(docker): clean up orphaned containers when docker run fails**. closed without merge
  - maintainer direction: category `silent_close`; implication `active`

## Fix status

- Merged to main in PR #39412 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-06-05`.
- Subsystem: `environments`