# NousResearch/hermes-agent · Vision image-source resolution and Docker backend patch
## Open now
- Nothing open on GitHub.
These items address a bug where the \`vision_analyze\` tool does not send images when using the Docker terminal backend, and introduce a unified image-source resolver to consolidate how image sources are handled across terminal backends.
## TL;DR
- 3 member(s): 0 open issue(s), 1 closed issue(s), 0 open PR(s), 1 merged PR(s), 1 PR(s) closed without merging.
- A fix was merged to main in PR #57890. Verify it addresses your case.
- **Recommended:** read PR #57890 (and any competing PRs below) before filing. Confirm the merged change covers your symptom.
## Maintainer resolution verdict
- Internal-tracker verdict over this cluster's issue members: `unresolved`. This is distinct from the fix status above. Weigh it yourself.
## How this cluster was grouped
- Link evidence (`STRUCTURAL_SPANNING`): every member is connected by structural evidence: shared changed files, or a closing/superseded reference.
- Pairwise link density `0.67`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked.
## Closed / duplicate issues
- #32709: Bug: \`vision_analyze\` tool fails to send images when using Docker terminal backend
## Prior & competing PRs
- **#57890: fix(vision): unified image-source resolver + terminal-backend confinement**. merged to main · fix tier `verified`
- maintainer direction: category `merged`
- files: `scripts/release.py`, `tests/integration/test_vision_docker_resolve.py`, `tests/tools/test_image_source.py`, `tests/tools/test_vision_native_fast_path.py`, `tests/tools/test_vision_tools.py`, `tools/credential_files.py`, `tools/image_source.py`, `tools/vision_tools.py`
- fix hunks:
- `scripts/release.py`
```diff
@@ -45,6 +45,7 @@
# Auto-extracted from noreply emails + manual overrides
AUTHOR_MAP = {
+ "[redacted:email]": "JAlmanzarMint", # PR #52688 salvage (vision: rasterize SVG / re-encode unsupported raster formats to PNG before embedding), folded into #57890
"[redacted:email]": "ooiuuii", # PR #58466/#58377 salvage (redact: fireworks fw-/fpk_ prefixes; telegram: redact bot tokens out of transport error strings)
"[redacted:email]": "msh01", # PR #58250 salvage (telegram: wall-clock init timeout via daemon-thread deadline + abandon the shielded initialize task on timeout so the retry ladder advances instead of hanging on attempt 1/8 under s6 supervision; #58236). Also covers PR #58276 salvage (compression: preserve a real user turn after compaction; #55677).
"[redacted:email]": "danilofalcao", # PR #56674 salvage (update: skip unsupported platform.matrix lazy refresh on native Windows — python-olm has no Windows wheel)
```
- `tests/integration/test_vision_docker_resolve.py`
```diff
@@ -0,0 +1,157 @@
+"""Docker integration tests for the vision image-source resolver.
+
+Exercises the in-sandbox exec-read path that unit tests can only mock. Two
+scenarios that ONLY the container filesystem can serve:
+
+ * a tmpfs ``/workspace`` file with no host path at all, and
+ * a root-owned mode-600 file the host user cannot read,
+
+both delivered by the resolver's ``base64`` exec-read fallback
+(``tools/image_source._resolve_container_fallback``). This is the same path
+that provides terminal-backend confinement for GHSA-gpxw-6wxv-w3qq: under a
+non-local backend a non-cache path is read INSIDE the container, never on the
+host.
+
+Gating follows the repo convention: the ``integration`` marker excludes these
+from the default suite (``addopts = -m 'not integration'`` in pyproject.toml);
+they run under ``pytest -m integration`` when a Docker daemon is available and
+skip cleanly when it is not. Container spin-up exceeds the 30s suite default,
+so the timeout is bumped to 180s.
+
+Run: pytest -m integration tests/integration/test_vision_docker_resolve.py
+"""
+import base64
+import shutil
+import subprocess
+
+import pytest
+
+
+def _docker_available() -> bool:
+ """True iff a docker CLI is on PATH and the daemon answers."""
+ if shutil.which("docker") is None:
+ return False
+ try:
+ return subprocess.run(
+ ["docker", "info"], capture_output=True, timeout=5
+ ).returncode == 0
+ except (subprocess.TimeoutExpired, OSError):
+ return False
+
+
+pytestmark = [
+ pytest.mark.integration,
+ pytest.mark.timeout(180),
+ pytest.mark.skipif(not _docker_available(), reason="Docker daemon not available"),
+]
+
+# A real 1x1 PNG.
+_TINY_PNG = base64.b64decode(
+ "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
+)
+
+
[redacted:email]
+def docker_backend(request, monkeypatch):
+ """A live DockerEnvironment registered so ``get_active_env`` finds it.
+
+ Unique task_id per test: DockerEnvironment derives the container name from
+ the task_id, so a shared id would make one test's teardown remove the
+ other's container.
+ """
+ from tools import terminal_tool
+ from tools.environments import docker as docker_env
+
+ # The resolver keys the exec-read off TERMINAL_ENV=docker.
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+
+ task_id = f"vision-docker-resolve-{request.node.name}"
+ env = docker_env.DockerEnvironment(
+ image="python:3.11-slim",
+ cwd="/workspace",
+ timeout=120,
+ task_id=task_id,
+ volumes=[],
+ network=False,
+ )
+ # Register under the raw task_id; get_active_env() falls back to the raw
+ # key after trying the _resolve_container_task_id() mapping.
+ with terminal_tool._env_lock:
+ terminal_tool._active_environments[task_id] = env
+ try:
+ env._task_id = task_id
+ yield env
+ finally:
+ with terminal_tool._env_lock:
+ terminal_tool._active_environments.pop(task_id, None)
+ try:
+ env.cleanup()
+ except Exception:
+ pass
+
+
+def _write_png_in_container(env, path, *, mode=None):
+ b64 = base64.b64encode(_TINY_PNG).decode()
+ cmd = f"printf %s {b64} | base64 -d > {path}"
+ if mode:
+ cmd += f" && chmod {mode} {path}"
+ res = env.execute(cmd)
+ assert res.get("returncode", 1) == 0, res
+
+
[redacted:email]
+async def test_resolves_tmpfs_workspace_file(docker_backend):
+ """A container-only path (no host file) is delivered via exec-read."""
+ from tools.image_source import ResolveContext, resolve_image_source
+
+ _write_png_in_container(docker_backend, "/workspace/shot.png")
+ res = await resolve_image_source(
+ "/workspace/shot.png", ResolveContext(task_id=docker_backend._task_id))
+ assert res.origin == "container"
+ assert res.mime == "image/png"
+ assert res.data == _TINY_PNG
+
+
[redacted:email]
+async def test_resolves_root_owned_mode600_file(docker_backend):
+ """Root-owned mode-600 (host user can't read it) is served in-container."""
+ from tools.image_source import ResolveContext, resolve_image_source
+
+ _write_png_in_container(docker_backend, "/workspace/secret.png", mode="600")
+ res = await resolve_image_source(
+ "/workspace/secret.png", ResolveContext(task_id=docker_backend._task_id))
+ assert res.origin == "container"
+ assert res.mime == "image/png"
+ assert res.data == _TINY_PNG
+
+
[redacted:email]
+async def test_host_secret_path_reads_container_not_host(docker_backend, tmp_path):
+ """The GHSA-gpxw invariant, end-to-end against real Docker.
+
+ A path that exists on the HOST with secret bytes but does NOT exist in the
+ container must resolve to the container read (which fails to find it) —
+ never to the host bytes. Proves vision cannot exfiltrate a host file under
+ a sandbox backend even when the exact path is real on the host.
+ """
+ from tools.image_source import (
+ ImageResolutionError,
+ ResolveContext,
+ resolve_image_source,
+ )
+
+ # A real host file outside any media cache, holding a secret.
+ host_secret = [redacted:secret-assignment] / "id_rsa"
+ host_secret.write_bytes(b"HOST-PRIVATE-KEY-DO-NOT-LEAK")
+
+ # That path does not exist inside the fresh container, so the exec-read
+ # finds nothing and the resolver refuses. The exact failure shape depends
+ # on the container's base64 build (some exit non-zero -> SourceNotFound,
+ # some print an error to stderr and exit 0 with empty stdout -> NotAnImage);
+ # both are ImageResolutionError. What matters for GHSA-gpxw: the resolver
+ # never returns the HOST bytes.
+ with pytest.raises(ImageResolutionError) as excinfo:
+ await resolve_image_source(
+ str(host_secret), ResolveContext(task_id=docker_backend._task_id))
+ # Belt and suspenders: the host secret must not appear anywhere in the error.
+ assert b"HOST-PRIVATE-KEY".decode() not in str(excinfo.value)
```
- `tests/tools/test_image_source.py`
```diff
@@ -0,0 +1,282 @@
+"""Tests for tools/image_source.py — the unified vision image-source resolver.
+
+Covers the delivery contract (data:/http/file/local/container source handling,
+size cap, magic-byte sniff) AND the terminal-backend confinement security model
+(GHSA-gpxw-6wxv-w3qq): under a non-local backend, host reads are confined to the
+media caches and every other path is read inside the sandbox via exec-read.
+"""
+
+import base64
+import importlib
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import patch
+
+import pytest
+
+
+PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 64
+JPEG = b"\xff\xd8\xff" + b"\x00" * 64
+
+
+def _reload(monkeypatch, hermes_home: Path):
+ monkeypatch.setenv("HERMES_HOME", str(hermes_home))
+ import hermes_constants
+ importlib.reload(hermes_constants)
+ import tools.image_source as isrc
+ importlib.reload(isrc)
+ return isrc
+
+
+class TestDataUrl:
+ @pytest.mark.asyncio
+ async def test_valid_data_url_resolves_to_bytes(self, tmp_path, monkeypatch):
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ b64 = base64.b64encode(PNG).decode()
+ res = await isrc.resolve_image_source(
+ f"data:image/png;base64,{b64}", isrc.ResolveContext())
+ assert res.data == PNG
+ assert res.mime == "image/png"
+ assert res.origin == "data"
+
+ @pytest.mark.asyncio
+ async def test_non_image_data_url_rejected(self, tmp_path, monkeypatch):
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ b64 = base64.b64encode(b"not an image").decode()
+ with pytest.raises(isrc.NotAnImage):
+ await isrc.resolve_image_source(
+ f"data:text/plain;base64,{b64}", isrc.ResolveContext())
+
+
+class TestLocalBackend:
+ @pytest.mark.asyncio
+ async def test_local_backend_reads_any_host_path(self, tmp_path, monkeypatch):
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ monkeypatch.setenv("TERMINAL_ENV", "local")
+ img = tmp_path / "outside" / "pic.png"
+ img.parent.mkdir(parents=True)
+ img.write_bytes(PNG)
+ res = await isrc.resolve_image_source(str(img), isrc.ResolveContext())
+ assert res.data == PNG
+ assert res.origin == "file"
+
+ @pytest.mark.asyncio
+ async def test_file_uri_scheme_stripped(self, tmp_path, monkeypatch):
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ monkeypatch.setenv("TERMINAL_ENV", "local")
+ img = tmp_path / "pic.jpg"
+ img.write_bytes(JPEG)
+ res = await isrc.resolve_image_source(f"file://{img}", isrc.ResolveContext())
+ assert res.mime == "image/jpeg"
+
+ @pytest.mark.asyncio
+ async def test_bare_relative_path_resolves(self, tmp_path, monkeypatch):
+ """A cwd-relative bare filename ('pic.png') is a valid local source —
+ main accepted it; the resolver must not regress it (PR review)."""
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ monkeypatch.setenv("TERMINAL_ENV", "local")
+ img = tmp_path / "pic.png"
+ img.write_bytes(PNG)
+ monkeypatch.chdir(tmp_path)
+ res = await isrc.resolve_image_source("pic.png", isrc.ResolveContext())
+ assert res.data == PNG
+ assert res.origin == "file"
+
+ @pytest.mark.asyncio
+ async def test_unknown_url_scheme_rejected(self, tmp_path, monkeypatch):
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ monkeypatch.setenv("TERMINAL_ENV", "local")
+ with pytest.raises(isrc.UnsupportedScheme):
+ await isrc.resolve_image_source(
+ "ftp://example.com/pic.png", isrc.ResolveContext())
+
+ @pytest.mark.asyncio
+ async def test_svg_passes_through_for_rasterization(self, tmp_path, monkeypatch):
+ """SVG has no raster magic bytes but is passed through with mime
+ image/svg+xml so the vision call sites can rasterize it to PNG."""
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ monkeypatch.setenv("TERMINAL_ENV", "local")
+ svg = tmp_path / "art.svg"
+ svg_bytes = b''
+ svg.write_bytes(svg_bytes)
+ res = await isrc.resolve_image_source(str(svg), isrc.ResolveContext())
+ assert res.mime == "image/svg+xml"
+ assert res.data == svg_bytes
+
+
+class TestNonLocalBackendConfinement:
+ """The security model: under a sandbox backend, host reads are confined to
+ the media caches; every other path is read inside the sandbox."""
+
+ @pytest.mark.asyncio
+ async def test_media_cache_path_host_read(self, tmp_path, monkeypatch):
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+ cached = home / "cache" / "images" / "inbound.png"
+ cached.parent.mkdir(parents=True)
+ cached.write_bytes(PNG)
+ # No sandbox env needed — a cache path is host-read directly.
+ res = await isrc.resolve_image_source(str(cached), isrc.ResolveContext())
+ assert res.data == PNG
+ assert res.origin == "file"
+
+ @pytest.mark.asyncio
+ async def test_host_secret_outside_cache_routes_to_sandbox_not_host(self, tmp_path, monkeypatch):
+ """A non-cache host path (e.g. /etc/passwd) must NOT be host-read — it
+ routes to the in-sandbox exec-read, which reads the CONTAINER's file."""
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+
+ # A real host file outside the caches, holding a "secret".
+ secret = [redacted:secret-assignment] / "id_rsa"
+ secret.write_bytes(b"HOST-PRIVATE-KEY-DO-NOT-LEAK")
+
+ # Fake sandbox env: its exec-read returns a *different* (container) image,
+ # proving we read the container filesystem, not the host secret.
+ container_png_b64 = base64.b64encode(PNG).decode()
+ calls = {}
+
+ def fake_execute(cmd, **kw):
+ calls["cmd"] = cmd
+ return {"returncode": 0, "output": container_png_b64}
+
+ with patch("tools.image_source._get_active_env",
+ return_value=SimpleNamespace(execute=fake_execute)):
+ res = await isrc.resolve_image_source(str(secret), isrc.ResolveContext(task_id="t1"))
+
+ # Read came from the sandbox exec-read, returning the container image —
+ # the host secret bytes never appear.
+ assert res.origin == "container"
+ assert res.data == PNG
+ assert b"HOST-PRIVATE-KEY" not in res.data
+ assert "head -c" in calls["cmd"] and "< " in calls["cmd"] # bounded, redirect-safe form
+
+ @pytest.mark.asyncio
+ async def test_non_cache_path_fails_closed_without_sandbox(self, tmp_path, monkeypatch):
+ """No active sandbox env -> refuse rather than fall back to a host read."""
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+ secret = [redacted:secret-assignment] / "id_rsa"
+ secret.write_bytes(b"HOST-PRIVATE-KEY")
+
+ with patch("tools.image_source._get_active_env", return_value=None):
+ with pytest.raises(isrc.SourceNotFound):
+ await isrc.resolve_image_source(str(secret), isrc.ResolveContext(task_id="t1"))
+
+ @pytest.mark.asyncio
+ async def test_symlink_in_cache_pointing_outside_is_not_host_read(self, tmp_path, monkeypatch):
+ """A symlink planted inside a cache dir that points at a host secret must
+ not be host-read (resolve() escapes the cache) — it routes to sandbox."""
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+ secret = [redacted:secret-assignment] / "outside" / "id_rsa"
+ secret.parent.mkdir(parents=True)
+ secret.write_bytes(b"HOST-PRIVATE-KEY")
+ cache_dir = home / "cache" / "images"
+ cache_dir.mkdir(parents=True)
+ link = cache_dir / "sneaky.png"
+ try:
+ link.symlink_to(secret)
+ except (OSError, NotImplementedError):
+ pytest.skip("symlinks unsupported")
+
+ # Fails closed (no sandbox) rather than host-reading the symlink target.
+ with patch("tools.image_source._get_active_env", return_value=None):
+ with pytest.raises(isrc.SourceNotFound):
+ await isrc.resolve_image_source(str(link), isrc.ResolveContext(task_id="t1"))
+
+
+class TestExecReadSafety:
+ @pytest.mark.asyncio
+ async def test_exec_read_is_bounded_and_redirect_safe(self, tmp_path, monkeypatch):
+ """Leading-dash paths go through an input redirect (no argv exposure)
+ and the read is size-bounded via head -c."""
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+ captured = {}
+
+ def fake_execute(cmd, **kw):
+ captured["cmd"] = cmd
+ return {"returncode": 0, "output": base64.b64encode(PNG).decode()}
+
+ with patch("tools.image_source._get_active_env",
+ return_value=SimpleNamespace(execute=fake_execute)):
+ await isrc.resolve_image_source(
+ "/workspace/-i-etc-shadow.png", isrc.ResolveContext(task_id="t1"))
+ assert f"head -c {isrc._MAX_INGEST_BYTES + 1} < " in captured["cmd"]
+ assert "'-i-etc-shadow.png'" in captured["cmd"] or "-i-etc-shadow.png" in captured["cmd"]
+
+ @pytest.mark.asyncio
+ async def test_exec_read_over_cap_rejected(self, tmp_path, monkeypatch):
+ """A sandbox file larger than the ingest cap is rejected, not embedded."""
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+ # head -c returns cap+1 bytes for an oversized file.
+ over = base64.b64encode(b"\x89PNG\r\n\x1a\n" + b"\x00" * (isrc._MAX_INGEST_BYTES - 7)).decode()
+
+ def fake_execute(cmd, **kw):
+ return {"returncode": 0, "output": over}
+
+ with patch("tools.image_source._get_active_env",
+ return_value=SimpleNamespace(execute=fake_execute)):
+ with pytest.raises(isrc.SourceTooLarge):
+ await isrc.resolve_image_source(
+ "/workspace/huge.png", isrc.ResolveContext(task_id="t1"))
+
+ @pytest.mark.asyncio
+ async def test_exec_read_nonzero_returncode_raises(self, tmp_path, monkeypatch):
+ home = tmp_path / "hermes"
+ isrc = _reload(monkeypatch, home)
+ monkeypatch.setenv("TERMINAL_ENV", "docker")
+
+ def fake_execute(cmd, **kw):
+ return {"returncode": 1, "output": ""}
+
+ with patch("tools.image_source._get_active_env",
+ return_value=SimpleNamespace(execute=fake_execute)):
+ with pytest.raises(isrc.SourceNotFound):
+ await isrc.resolve_image_source(
+ "/workspace/nope.png", isrc.ResolveContext(task_id="t1"))
+
+
+class TestSvgNormalization:
+ """SVG resolves end-to-end: the resolver passes it through as
+ image/svg+xml and the vision call sites rasterize it to PNG via
+ _normalize_to_supported_image (PR #52688, folded in)."""
+
+ @pytest.mark.asyncio
+ async def test_svg_rasterized_when_converter_available(self, tmp_path, monkeypatch):
+ from tools import vision_tools as vt
+ isrc = _reload(monkeypatch, tmp_path / "hermes")
+ monkeypatch.setenv("TERMINAL_ENV", "local")
+ svg = tmp_path / "art.svg"
+ svg.write_bytes(b'')
+
+ def fake_rasterize(svg_path, out_path):
+ out_path.write_bytes(PNG)
+ return True
+
+ with patch.object(vt, "_rasterize_svg_to_png", side_effect=fake_rasterize):
+ res = await isrc.resolve_image_source(str(svg), isrc.ResolveContext())
+ assert res.mime == "image/svg+xml"
+ path, mime, err = vt._normalize_to_supported_image(svg, "image/svg+xml")
+ assert err is None
+ assert mime == "image/png"
+ assert path.read_bytes() == PNG
+ path.unlink()
+
+ def test_svg_actionable_error_when_no_converter(self, tmp_path, monkeypatch):
+ from tools import vision_tools as vt
+ _reload(monkeypatch, tmp_path / "hermes")
+ svg = tmp_path / "art.svg"
+ svg.write_bytes(b'')
+ with patch.object(vt, "_rasterize_svg_to_png", return_value=False):
+ path, mime, err = vt._normalize_to_supported_image(svg, "image/svg+xml")
+ assert path is None
+ assert "rasterizer" in err
```
- `tests/tools/test_vision_native_fast_path.py`
```diff
@@ -119,7 +119,9 @@ def test_missing_file_returns_error_string(self, tmp_path):
assert isinstance(result, str)
parsed = json.loads(result)
assert parsed.get("success") is False
- assert "Invalid image source" in parsed.get("error", "")
+ # Unified resolver: local backend reports a clean not-found.
+ err = parsed.get("error", "").lower()
+ assert "image file not found" in err or "no active sandbox" in err
def test_empty_image_url_returns_error(self):
result = asyncio.get_event_loop().run_until_complete(
```
- `tests/tools/test_vision_tools.py`
```diff
@@ -1,5 +1,6 @@
"""Tests for tools/vision_tools.py — URL validation, type hints, error logging."""
+import base64
import json
import logging
import os
```
```diff
@@ -381,26 +382,20 @@ async def test_analysis_error_logs_exc_info(self, caplog):
@pytest.mark.asyncio
async def test_cleanup_error_logs_exc_info(self, tmp_path, caplog):
"""Temp file cleanup failure should log warning with exc_info."""
- # Create a real temp file that will be "downloaded"
- temp_dir = tmp_path / "temp_vision_images"
- temp_dir.mkdir()
-
- async def fake_download(url, dest, max_retries=3):
- """Simulate download by writing file to the expected destination."""
- dest.parent.mkdir(parents=True, exist_ok=True)
- dest.write_bytes(b"\xff\xd8\xff" + b"\x00" * 16)
- return dest
+ # A data: URL resolves to bytes without any network, materializes to a
+ # vision temp file, then the analysis runs — exercising the temp-cleanup
+ # path in the finally block. A tiny valid JPEG (magic bytes) passes the
+ # resolver's magic-byte sniff.
+ jpeg_b64 = base64.b64encode(b"\xff\xd8\xff" + b"\x00" * 32).decode("ascii")
+ data_url = f"data:image/jpeg;base64,{jpeg_b64}"
with (
- patch("tools.vision_tools._validate_image_url_async", new_callable=AsyncMock, return_value=True),
- patch("tools.vision_tools._download_image", side_effect=fake_download),
patch(
"tools.vision_tools._image_to_base64_data_url",
return_value="data:image/jpeg;base64,abc",
),
caplog.at_level(logging.WARNING, logger="tools.vision_tools"),
):
- # Mock the async_call_llm function to return a mock response
mock_response = MagicMock()
mock_choice = MagicMock()
mock_choice.message.content = "A test image description"
```
```diff
@@ -409,16 +404,12 @@ async def fake_download(url, dest, max_retries=3):
with (
patch("tools.vision_tools.async_call_llm", new_callable=AsyncMock, return_value=mock_response),
):
- # Make unlink fail to trigger cleanup warning
- original_unlink = Path.unlink
-
+ # Make unlink fail to trigger the cleanup warning.
def failing_unlink(self, *args, **kwargs):
raise PermissionError("no permission")
with patch.object(Path, "unlink", failing_unlink):
- result = await vision_analyze_tool(
- "https://example.com/tempimg.jpg", "describe", "test/model"
- )
+ result = await vision_analyze_tool(data_url, "describe", "test/model")
warning_records = [
r
```
```diff
@@ -500,7 +491,8 @@ async def test_local_non_image_file_rejected_before_llm_call(self, tmp_path):
result = json.loads(await vision_analyze_tool(str(secret), "extract text"))
assert result["success"] is False
- assert "Only real image files are supported" in result["error"]
+ # The unified resolver's magic-byte sniff rejects non-images.
+ assert "not a recognized image" in result["error"]
mock_llm.assert_not_awaited()
@pytest.mark.asyncio
```
```diff
@@ -513,8 +505,8 @@ async def test_blocked_remote_url_short_circuits_before_download(self):
}
with (
- patch("tools.vision_tools.check_website_access", return_value=blocked),
- patch("tools.vision_tools._validate_image_url_async", new_callable=AsyncMock, return_value=True),
+ patch("tools.website_policy.check_website_access", return_value=blocked),
+ patch("tools.url_safety.is_safe_url", return_value=True),
patch("tools.vision_tools._download_image", new_callable=AsyncMock) as mock_download,
):
result = json.loads(await vision_analyze_tool("https://blocked.test/cat.png", "describe"))
```
```diff
@@ -1258,7 +1250,7 @@ def slow_encode(path, mime_type=None):
enc_inflight -= 1
return "data:image/jpeg;base64,AAAA"
- async def fake_native(image_url, question):
+ async def fake_native(image_url, question, task_id=None):
nonlocal calls_inflight, calls_peak
calls_inflight += 1
calls_peak = max(calls_peak, calls_inflight)
```
- `tools/credential_files.py`
```diff
@@ -402,6 +402,30 @@ def map_cache_path_to_container(
return None
+def from_agent_visible_cache_path(
+ container_path: str,
+ container_base: str = "/root/.hermes",
+) -> str:
+ """Translate a sandbox/container cache path back to its host path.
+
+ Inverse of :func:`to_agent_visible_cache_path`. Returns the input unchanged
+ when the active backend is not Docker, or when the path is not under any
+ auto-mounted cache directory — the caller then treats a still-container
+ path as "no host file" and falls back to an in-container read.
+ """
+ if os.environ.get("TERMINAL_ENV", "local") != "docker":
+ return container_path
+
+ path = Path(container_path)
+ for mount in get_cache_directory_mounts(container_base=container_base):
+ try:
+ rel = path.relative_to(mount["container_path"])
+ except ValueError:
+ continue
+ return str(Path(mount["host_path"]) / rel)
+ return container_path
+
+
def to_agent_visible_cache_path(
host_path: str,
container_base: str = "/root/.hermes",
```
- `tools/image_source.py`
```diff
@@ -0,0 +1,322 @@
+"""Single resolver for every vision_analyze image source -> bytes + mime.
+
+All source handling (data:/http(s)/file/local/container) funnels through
+:func:`resolve_image_source` so size and magic-byte checks are enforced exactly
+once. Returns raw bytes (not a path): the downstream step is base64 -> data URL
+(RFC 2397) and provider base64 content blocks.
+
+Security (terminal-backend confinement, GHSA-gpxw-6wxv-w3qq): under a non-local
+terminal backend the file tools are confined to the sandbox (SECURITY.md 2.2),
+but vision read images host-side. This resolver enforces the same boundary:
+
+ * local backend -> read any host path (chosen posture, unchanged)
+ * non-local backend:
+ path in a media cache -> host-read (the gateway/download caches live on
+ the host and are bind-mounted into the sandbox)
+ path anywhere else -> read the bytes *inside the sandbox* via exec-read
+ (the agent can already ``cat`` any container file;
+ this stays within the sandbox boundary and never
+ reaches the host's ``/etc/passwd`` / ``~/.ssh``).
+
+So a prompt-injected ``vision_analyze('/etc/passwd')`` under Docker reads the
+*container's* file (what every other tool sees), not the host's — no escape —
+while container-only images (tmpfs ``/workspace``, root-owned) are still
+deliverable. This is the unified delivery + confinement model: the same
+mechanism that fixes "vision can't see container files" also closes the escape.
+"""
+from __future__ import annotations
+
+import asyncio
+import base64
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional
+
+# Raw-bytes INGEST budget — what the resolver will load before handing off.
+# This is deliberately the 50MB download cap (tools/vision_tools._VISION_MAX_DOWNLOAD_BYTES),
+# NOT the 20MB provider payload cap. The 20MB cap (_MAX_BASE64_BYTES) is a
+# *post-resize* limit enforced at the call sites: an oversized raw image must
+# still reach the resizer so it can be downscaled under the payload cap. Capping
+# raw bytes at 20MB here would reject every 20-50MB photo before resize can run.
+_MAX_INGEST_BYTES = 50 * 1024 * 1024
+
+
+class ImageResolutionError(Exception):
+ def __init__(self, message: str, *, src: str = "", origin: str = ""):
+ super().__init__(message)
+ self.src, self.origin = src, origin
+
+
+class UnsupportedScheme(ImageResolutionError):
+ pass
+
+
+class SourceUnsafe(ImageResolutionError): # SSRF / path-allowlist
+ pass
+
+
+class SourceTooLarge(ImageResolutionError):
+ pass
+
+
+class SourceNotFound(ImageResolutionError):
+ pass
+
+
+class NotAnImage(ImageResolutionError):
+ pass
+
+
+@dataclass
+class ResolveContext:
+ task_id: Optional[str] = None
+
+
+@dataclass
+class ResolvedImage:
+ data: bytes
+ mime: str
+ origin: str # one of: data | http | file | local | container
+
+
+# Explicit URL scheme, e.g. "ftp://", "s3://". Bare Windows drive paths
+# ("C:\x.png") don't match because they lack the "//".
+_SCHEME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*://")
+
+
+async def resolve_image_source(src: str, ctx: ResolveContext) -> ResolvedImage:
+ if not isinstance(src, str) or not src.strip():
+ raise SourceNotFound("image_url is required", src=str(src))
+ s = src.strip()
+ if s.startswith("data:"):
+ data, mime = _resolve_data_url(s)
+ return _finalize(data, mime, "data", s)
+ if s.startswith(("http://", "https://")):
+ reason = _http_block_reason(s)
+ if reason:
+ raise SourceUnsafe(reason, src=s)
+ return _finalize(await _download_to_bytes(s), "", "http", s)
+
+ if _SCHEME_RE.match(s) and not s.lower().startswith("file://"):
+ raise UnsupportedScheme(
+ "Unrecognized image source scheme. Use an http(s) URL, a local "
+ "file path, a file:// URI, or a data: URL.",
+ src=s,
+ )
+
+ # Everything else is a filesystem path — including bare relative names
+ # like "pic.png" (accepted on main; a path-shape gate here regressed them).
+ candidate = s[len("file://"):] if s.lower().startswith("file://") else s
+ p = Path(os.path.expanduser(candidate))
+ # Confinement decision (see module docstring). Under a non-local backend
+ # a path is host-readable ONLY if it lands in a media cache (after
+ # translating a container-visible cache path back to its host mount);
+ # every other path is read inside the sandbox via exec-read, so a host
+ # path outside the caches never yields the host's bytes.
+ host_target = _permitted_host_read_target(p, ctx)
+ if host_target is not None and host_target.is_file():
+ data = await asyncio.to_thread(host_target.read_bytes)
+ return _finalize(data, "", "file", s)
+ if _is_local_terminal_backend():
+ # Local backend: any path was host-readable, so a miss simply means
+ # the file doesn't exist — no sandbox to fall back to.
+ raise SourceNotFound(f"image file not found: '{p}'", src=s, origin="file")
+ # Not a permitted host read (or the host file is absent) -> read the
+ # bytes inside the sandbox. Under a sandbox this reads the container's
+ # filesystem, never the host's.
+ return await _resolve_container_fallback(p, ctx, s)
+
+
+def _resolve_data_url(s: str) -> tuple[bytes, str]:
+ header, _, payload = s.partition(",")
+ if ";base64" not in header:
+ raise NotAnImage("data: URL must be base64-encoded", src=s[:64])
+ declared = header[len("data:"):].split(";", 1)[0].strip() or "application/octet-stream"
+ # Cheap pre-decode size gate on the encoded length (~4/3 expansion).
+ if (len(payload) * 3) // 4 > _MAX_INGEST_BYTES:
+ raise SourceTooLarge("data: URL exceeds size limit", src=s[:64])
+ try:
+ data = base64.b64decode(payload, validate=True)
+ except Exception as exc:
+ raise NotAnImage(f"invalid base64 in data: URL: {exc}", src=s[:64])
+ return data, declared # real mime verified in _finalize via magic bytes
+
+
+def _http_block_reason(url: str) -> Optional[str]:
+ """Return a human-readable block reason, or None when the URL is allowed.
+
+ Pre-flight short-circuit: policy-blocked URLs are refused BEFORE any
+ network I/O. ``_download_image`` re-checks policy internally (per attempt
+ and against the final redirect target) — that second evaluation is
+ intentional, not redundant: this one guarantees no bytes move for a
+ blocked URL; the inner one covers redirects and non-resolver callers.
+ Preserves the specific website-policy message so the agent sees *why*.
+ """
+ from tools.url_safety import is_safe_url
+ from tools.website_policy import check_website_access
+
+ if not is_safe_url(url):
+ return "blocked: unsafe or private URL"
+ blocked = check_website_access(url)
+ if blocked:
+ return blocked.get("message") or "blocked by website policy"
+ return None
+
+
+async def _download_to_bytes(url: str) -> bytes:
+ import tempfile
+
+ from tools.vision_tools import _download_image
+
+ with tempfile.NamedTemporaryFile(suffix=".img", delete=False) as tf:
+ tmp = Path(tf.name)
+ try:
+ # Enforces the 50MB stream cap, redirect SSRF guard, and website policy.
+ await _download_image(url, tmp)
+ return await asyncio.to_thread(tmp.read_bytes)
+ except PermissionError as exc: # website policy block
+ raise SourceUnsafe(str(exc), src=url, origin="http")
+ finally:
+ tmp.unlink(missing_ok=True)
+
+
+def _is_local_terminal_backend() -> bool:
+ """True when the terminal backend runs directly on the host.
+
+ Mirrors ``tools.browser_tool._is_local_backend`` and terminal_tool's own
+ dispatch, which key off ``TERMINAL_ENV``.
+ """
+ return os.getenv("TERMINAL_ENV", "local").strip().lower() in ("local", "")
+
+
+def _media_cache_roots() -> list:
+ """Agent-managed media cache directories under HERMES_HOME (host side).
+
+ The only host paths vision may read under a non-local backend: gateway-
+ downloaded inbound media and the tools' own URL-download temp dirs. Covers
+ the consolidated ``cache/`` layout and the legacy flat directories.
+ """
+ from hermes_constants import get_hermes_home
+
+ home = get_hermes_home()
+ return [
+ home / "cache", # cache/images, cache/vision, cache/video(s), cache/audio
+ home / "image_cache",
+ home / "audio_cache",
+ home / "video_cache",
+ home / "temp_vision_images",
+ home / "temp_video_files",
+ ]
+
+
+def _permitted_host_read_target(p: Path, ctx: ResolveContext) -> Optional[Path]:
+ """Return the host path to read, or ``None`` if a host read is not permitted.
+
+ - Local backend: any path is permitted (chosen posture). Returns ``p``.
+ - Non-local backend: permitted only if the path resolves inside a media
+ cache root. A container-visible cache path (e.g. ``/root/.hermes/cache/
+ images/x.png``) is first translated back to its host mount; anything that
+ is not under a cache returns ``None`` so the caller routes it to the
+ in-sandbox exec-read instead of reading the host filesystem.
+ """
+ if _is_local_terminal_backend():
+ try:
+ return p.resolve()
+ except Exception: # noqa: BLE001 — unresolved path: let is_file() fail downstream
+ return p
+
+ from tools.credential_files import from_agent_visible_cache_path
+
+ host_candidate = Path(from_agent_visible_cache_path(str(p)))
+ try:
+ real = host_candidate.resolve()
+ except Exception: # noqa: BLE001 — cannot resolve -> not a safe host read
+ return None
+ for root in _media_cache_roots():
+ try:
+ real.relative_to(root.resolve())
+ return real
+ except ValueError:
+ continue
+ return None
+
+
+def _get_active_env(task_id: Optional[str]):
+ if not task_id:
+ return None
+ try:
+ from tools.terminal_tool import get_active_env
+
+ return get_active_env(task_id)
+ except Exception:
+ return None
+
+
+async def _resolve_container_fallback(p: Path, ctx: ResolveContext, src: str) -> ResolvedImage:
+ """Read the image bytes inside the sandbox (fail-closed when none exists).
+
+ Reached when a host read is not permitted or the host file is absent. The
+ agent can already ``cat`` any container file (file_operations.py reads
+ root-owned mode-600 files this way), so this stays within the same sandbox
+ boundary and never touches the host filesystem. ``--`` stops a leading-dash
+ path from being parsed as a ``base64`` option; ``base64 -w0`` is GNU-only,
+ so pipe through ``tr -d`` for BusyBox.
+
+ Fail-closed: if there is no active sandbox env we refuse rather than falling
+ back to a host read, so a non-cache host path under a sandbox never leaks.
+ """
+ import asyncio
+ import shlex
+
+ env = _get_active_env(ctx.task_id)
+ if env is None:
+ raise SourceNotFound(
+ f"'{p}' is not reachable inside the sandbox and no active sandbox "
+ f"session is available to read it",
+ src=src, origin="container")
+
+ # Bound the read INSIDE the sandbox: head -c caps at ingest-limit+1 bytes
+ # so a huge file (or /dev/zero) can't stream unbounded base64 into host
+ # memory — the +1 byte lets us distinguish "exactly at the cap" from
+ # "over the cap" after decode. The input redirect (< path) avoids argv
+ # entirely, so leading-dash paths can't be parsed as options; base64
+ # -w0 is GNU-only, so pipe through tr -d for BusyBox.
+ # env.execute is a blocking backend exec; keep it off the event loop so a
+ # multi-MB base64 read doesn't stall every other coroutine.
+ qp = shlex.quote(str(p))
+ res = await asyncio.to_thread(
+ env.execute,
+ f"head -c {_MAX_INGEST_BYTES + 1} < {qp} | base64 | tr -d '\\n'")
+ if res.get("returncode", 1) != 0:
+ raise SourceNotFound(f"could not read '{p}' inside the sandbox", src=src, origin="container")
+ try:
+ data = base64.b64decode(res.get("output", ""), validate=True)
+ except Exception as exc:
+ raise NotAnImage(f"sandbox returned non-image data for '{p}': {exc}", src=src)
+ if len(data) > _MAX_INGEST_BYTES:
+ raise SourceTooLarge("image exceeds size limit", src=src, origin="container")
+ return _finalize(data, "", "container", src)
+
+
+def _finalize(data: bytes, declared_mime: str, origin: str, src: str) -> ResolvedImage:
+ """Intrinsic-correctness chokepoint: ingest byte cap + magic-byte sniff.
+
+ The cap here is the generous 50MB *ingest* budget, not the 20MB provider
+ payload cap — a 20-50MB image must survive this step so the call site can
+ resize it under the payload cap. See ``_MAX_INGEST_BYTES``.
+ """
+ from tools.vision_tools import _detect_image_mime_type_from_bytes
+
+ if len(data) > _MAX_INGEST_BYTES:
+ raise SourceTooLarge("image exceeds size limit", src=src, origin=origin)
+ sniffed = _detect_image_mime_type_from_bytes(data)
+ if sniffed is None:
+ if b" bool:
return await async_is_safe_url(url)
-def _detect_image_mime_type(image_path: Path) -> Optional[str]:
- """Return a MIME type when the file looks like a supported image."""
- with image_path.open("rb") as f:
- header = f.read(64)
+def _detect_image_mime_type_from_bytes(data: bytes) -> Optional[str]:
+ """Magic-byte MIME sniff on raw bytes (authoritative; no extension trust).
+ Returns ``None`` for anything without a recognized image header — including
+ SVG, which has no magic bytes. The resolver special-cases SVG (sniffs
+ `` Optional[str]:
return "image/bmp"
if len(header) >= 12 and header[:4] == b"RIFF" and header[8:12] == b"WEBP":
return "image/webp"
- if image_path.suffix.lower() == ".svg":
- head = image_path.read_text(encoding="utf-8", errors="ignore")[:4096].lower()
- if " bool:
+ """Best-effort SVG → PNG rasterization. Returns True on success.
+
+ Tries, in order: cairosvg, svglib+reportlab, then system rasterizers
+ (rsvg-convert, inkscape). All are soft dependencies; if none is available
+ we return False and the caller rejects the image with an actionable error
+ rather than embedding an unsupported media_type that would wedge the
+ session.
+ """
+ # 1) cairosvg (pure-python-ish, most common)
+ try:
+ import cairosvg # type: ignore
+ cairosvg.svg2png(url=str(svg_path), write_to=str(out_path))
+ return out_path.exists() and out_path.stat().st_size > 0
+ except Exception:
+ pass
+ # 2) svglib + reportlab
+ try:
+ from svglib.svglib import svg2rlg # type: ignore
+ from reportlab.graphics import renderPM # type: ignore
+ drawing = svg2rlg(str(svg_path))
+ if drawing is not None:
+ renderPM.drawToFile(drawing, str(out_path), fmt="PNG")
+ return out_path.exists() and out_path.stat().st_size > 0
+ except Exception:
+ pass
+ # 3) system rasterizers
+ import shutil as _shutil
+ import subprocess as _subprocess
+ for cmd in (
+ ["rsvg-convert", "-o", str(out_path), str(svg_path)],
+ ["inkscape", str(svg_path), "--export-type=png",
+ f"--export-filename={out_path}"],
+ ):
+ if _shutil.which(cmd[0]):
+ try:
+ _subprocess.run(
+ cmd, check=True, capture_output=True, timeout=30,
+ stdin=_subprocess.DEVNULL,
+ )
+ if out_path.exists() and out_path.stat().st_size > 0:
+ return True
+ except Exception:
+ continue
+ return False
+
+
+def _normalize_to_supported_image(
+ image_path: Path, detected_mime: str
+) -> tuple[Optional[Path], Optional[str], Optional[str]]:
+ """Ensure an image is in a vision-provider-supported format.
+
+ Returns a 3-tuple ``(path, mime, error)``:
+ - If ``detected_mime`` is already supported: ``(image_path, detected_mime, None)``.
+ - If conversion succeeds: ``(new_png_path, "image/png", None)`` — the new
+ path is a temp file the CALLER must clean up.
+ - If conversion is impossible: ``(None, None, )``.
+
+ SVG is rasterized to PNG (best-effort, soft deps). Other raster formats
+ Pillow can read (BMP, TIFF, etc.) are re-encoded to PNG. This runs BEFORE
+ the image is base64-embedded into conversation history, so an unsupported
+ media_type can never reach the provider and wedge the session.
+ """
+ if detected_mime in _ANTHROPIC_SUPPORTED_MEDIA_TYPES:
+ return image_path, detected_mime, None
+
+ out_dir = get_hermes_dir("cache/vision", "temp_vision_images")
+ out_dir.mkdir(parents=True, exist_ok=True)
+ out_path = out_dir / f"converted_{uuid.uuid4()}.png"
+
+ # SVG: needs a rasterizer (Pillow cannot render SVG).
+ if detected_mime == "image/svg+xml":
+ if _rasterize_svg_to_png(image_path, out_path):
+ return out_path, "image/png", None
+ return (
+ None,
+ None,
+ "This is an SVG, which vision models cannot read directly, and no "
+ "SVG rasterizer is installed (tried cairosvg, svglib, rsvg-convert, "
+ "inkscape). Convert the SVG to PNG first — e.g. open it in a browser "
+ "and screenshot it, or install a rasterizer "
+ "(`pip install cairosvg`) — then re-run vision_analyze on the PNG.",
+ )
+
+ # Other non-supported raster formats (BMP, TIFF, ...): re-encode via Pillow.
+ try:
+ from PIL import Image as _PILImage
+ with _PILImage.open(image_path) as _img:
+ if _img.mode not in ("RGB", "RGBA", "L"):
+ _img = _img.convert("RGBA")
+ _img.save(out_path, format="PNG")
+ if out_path.exists() and out_path.stat().st_size > 0:
+ return out_path, "image/png", None
+ except Exception as _exc:
+ logger.warning("Failed to normalize %s image to PNG: %s",
+ detected_mime, _exc)
+ return (
+ None,
+ None,
+ f"Image format {detected_mime!r} is not supported by the vision API "
+ f"and could not be converted to PNG (install Pillow for raster "
+ f"conversion). Convert it to PNG or JPEG and try again.",
+ )
+
+
def _is_retryable_download_error(error: Exception) -> bool:
"""Return True only for transient image-download failures worth retrying.
```
```diff
@@ -817,13 +932,14 @@ async def _vision_concurrency_slot():
async def _vision_analyze_native(
image_url: str,
question: str,
+ task_id: Optional[str] = None,
) -> Any:
"""Fast path for vision-capable main models.
- Loads the image (local file OR remote URL), base64-encodes it, and
- returns a multimodal tool-result envelope. The agent loop unwraps it;
- provider adapters serialize it into the right tool-result-with-image
- shape for each backend.
+ Loads the image (data: / http(s) / file:// / local path / sandbox-container
+ path) via the unified resolver, base64-encodes it, and returns a multimodal
+ tool-result envelope. The agent loop unwraps it; provider adapters serialize
+ it into the right tool-result-with-image shape for each backend.
Returns:
A ``_multimodal`` envelope dict on success.
```
```diff
@@ -840,38 +956,51 @@ async def _vision_analyze_native(
if is_interrupted():
return tool_error("Interrupted", success=False)
- # Resolve the image source (mirrors vision_analyze_tool's logic
- # exactly so behaviour is consistent).
- resolved_url = image_url
- if resolved_url.startswith("file://"):
- resolved_url = resolved_url[len("file://"):]
- local_path = Path(os.path.expanduser(resolved_url))
-
- if local_path.is_file():
- temp_image_path = local_path
- should_cleanup = False
- elif await _validate_image_url_async(image_url):
- blocked = check_website_access(image_url)
- if blocked:
- return tool_error(blocked["message"], success=False)
- temp_dir = get_hermes_dir("cache/vision", "temp_vision_images")
- temp_image_path = temp_dir / f"temp_image_{uuid.uuid4()}.jpg"
- await _download_image(image_url, temp_image_path)
- should_cleanup = True
- else:
- return tool_error(
- "Invalid image source. Provide an HTTP/HTTPS URL or a "
- "valid local file path.",
- success=False,
- )
+ # Resolve the source to raw bytes through the single resolver (unifies
+ # data:/http/file/local/container and enforces terminal-backend
+ # confinement). Materialize to a temp file so the existing path-based
+ # encode/resize/embed-cap pipeline below is reused verbatim.
+ from tools.image_source import (
+ ImageResolutionError,
+ ResolveContext,
+ resolve_image_source,
+ )
- image_size_bytes = temp_image_path.stat().st_size
- detected_mime_type = _detect_image_mime_type(temp_image_path)
- if not detected_mime_type:
+ try:
+ resolved = await resolve_image_source(image_url, ResolveContext(task_id=task_id))
+ except ImageResolutionError as exc:
+ return tool_error(str(exc), success=False)
+
+ detected_mime_type = resolved.mime
+ image_size_bytes = len(resolved.data)
+ temp_dir = get_hermes_dir("cache/vision", "temp_vision_images")
+ temp_dir.mkdir(parents=True, exist_ok=True)
+ temp_image_path = temp_dir / f"temp_image_{uuid.uuid4()}.img"
+ await asyncio.to_thread(temp_image_path.write_bytes, resolved.data)
+ should_cleanup = True
+
+ # Normalize unsupported formats (SVG, BMP, ...) to PNG BEFORE embedding.
+ # Anthropic only accepts jpeg/png/gif/webp; an unsupported media_type
+ # baked into immutable history wedges the session with a 400 on every
+ # resume. Convert here so it can never enter history. Offloaded — the
+ # rasterizers/Pillow are blocking.
+ normalized_path, detected_mime_type, _norm_err = await asyncio.to_thread(
+ _normalize_to_supported_image, temp_image_path, detected_mime_type,
+ )
+ if _norm_err or normalized_path is None:
return tool_error(
- "Only real image files are supported for vision analysis.",
- success=False,
+ _norm_err or "Image normalization failed.", success=False,
)
+ if normalized_path != temp_image_path:
+ # We created a temp PNG — swap to it and ensure it's cleaned up.
+ if should_cleanup and temp_image_path.exists():
+ try:
+ temp_image_path.unlink()
+ except Exception:
+ pass
+ temp_image_path = normalized_path
+ should_cleanup = True
+ image_size_bytes = temp_image_path.stat().st_size
image_data_url = await _run_encode_on_cpu_executor(
_image_to_base64_data_url,
```
```diff
@@ -935,6 +1064,7 @@ async def vision_analyze_tool(
image_url: str,
user_prompt: str,
model: str = None,
+ task_id: Optional[str] = None,
) -> str:
"""
Analyze an image from a URL or local file path using vision AI.
```
```diff
@@ -996,42 +1126,50 @@ async def vision_analyze_tool(
logger.info("Analyzing image: %s", image_url[:60])
logger.info("User prompt: %s", user_prompt[:100])
-
- # Determine if this is a local file path or a remote URL
- # Strip file:// scheme so file URIs resolve as local paths.
- resolved_url = image_url
- if resolved_url.startswith("file://"):
- resolved_url = resolved_url[len("file://"):]
- local_path = Path(os.path.expanduser(resolved_url))
- if local_path.is_file():
- # Local file path (e.g. from platform image cache) -- skip download
- logger.info("Using local image file: %s", image_url)
- temp_image_path = local_path
- should_cleanup = False # Don't delete cached/local files
- elif await _validate_image_url_async(image_url):
- # Remote URL -- download to a temporary location
- blocked = check_website_access(image_url)
- if blocked:
- raise PermissionError(blocked["message"])
- logger.info("Downloading image from URL...")
- temp_dir = get_hermes_dir("cache/vision", "temp_vision_images")
- temp_image_path = temp_dir / f"temp_image_{uuid.uuid4()}.jpg"
- await _download_image(image_url, temp_image_path)
- should_cleanup = True
- else:
- raise ValueError(
- "Invalid image source. Provide an HTTP/HTTPS URL or a valid local file path."
- )
-
+
+ # Resolve the source to raw bytes through the single resolver (unifies
+ # data:/http/file/local/container and enforces terminal-backend
+ # confinement). Materialize to a temp file so the existing path-based
+ # encode/resize pipeline below is reused verbatim.
+ from tools.image_source import (
+ ImageResolutionError,
+ ResolveContext,
+ resolve_image_source,
+ )
+
+ try:
+ resolved = await resolve_image_source(image_url, ResolveContext(task_id=task_id))
+ except ImageResolutionError as exc:
+ raise ValueError(str(exc))
+
+ detected_mime_type = resolved.mime
+ temp_dir = get_hermes_dir("cache/vision", "temp_vision_images")
+ temp_dir.mkdir(parents=True, exist_ok=True)
+ temp_image_path = temp_dir / f"temp_image_{uuid.uuid4()}.img"
+ await asyncio.to_thread(temp_image_path.write_bytes, resolved.data)
+ should_cleanup = True
+
# Get image file size for logging
- image_size_bytes = temp_image_path.stat().st_size
+ image_size_bytes = len(resolved.data)
image_size_kb = image_size_bytes / 1024
logger.info("Image ready (%.1f KB)", image_size_kb)
+ # Normalize unsupported formats (SVG, BMP, ...) to PNG. Vision providers
+ # reject these media types; convert before encoding. Offloaded — the
+ # rasterizers/Pillow are blocking.
+ normalized_path, detected_mime_type, _norm_err = await asyncio.to_thread(
+ _normalize_to_supported_image, temp_image_path, detected_mime_type,
+ )
+ if _norm_err or normalized_path is None:
+ raise ValueError(_norm_err or "Image normalization failed.")
+ if normalized_path != temp_image_path:
+ if should_cleanup and temp_image_path.exists():
+ try:
+ temp_image_path.unlink()
+ except Exception:
+ pass
+ temp_image_path = normalized_path
+ should_cleanup = True
- detected_mime_type = _detect_image_mime_type(temp_image_path)
- if not detected_mime_type:
- raise ValueError("Only real image files are supported for vision analysis.")
-
# Convert image to base64 — send at full resolution first.
# If the provider rejects it as too large, we auto-resize and retry.
# Offloaded to the bounded vision CPU executor so a fan-out of encodes
```
```diff
@@ -1335,6 +1473,7 @@ def check_vision_requirements() -> bool:
async def _handle_vision_analyze(args: Dict[str, Any], **kw: Any) -> str:
image_url = args.get("image_url", "")
question = args.get("question", "")
+ task_id = kw.get("task_id")
# The fan-out cap lives inside the encode/resize step (offloaded to the
# bounded _vision_cpu_executor), NOT around the whole analysis — so a
```
```diff
@@ -1349,7 +1488,7 @@ async def _handle_vision_analyze(args: Dict[str, Any], **kw: Any) -> str:
# information loss, no extra latency.
if _should_use_native_vision_fast_path():
logger.info("vision_analyze: native fast path")
- return await _vision_analyze_native(image_url, question)
+ return await _vision_analyze_native(image_url, question, task_id=task_id)
# Legacy path: aux LLM describes the image and we return its text.
full_prompt = (
```
```diff
@@ -1368,7 +1507,7 @@ async def _handle_vision_analyze(args: Dict[str, Any], **kw: Any) -> str:
pass
if not model:
model = os.getenv("AUXILIARY_VISION_MODEL", "").strip() or None
- return await vision_analyze_tool(image_url, full_prompt, model)
+ return await vision_analyze_tool(image_url, full_prompt, model, task_id=task_id)
registry.register(
```
- **#35362: fix(vision): unify vision_analyze image-source resolution through one resolver**. closed without merge
- maintainer direction: category `merged_via`; implication `superseded`; refs `#57890`
## Fix status
- Merged to main in PR #57890 (cited above). No release data is tracked. Verify downstream.
- Fix last verified (merged) `2026-07-04`.
- Subsystem: `agent`