# NousResearch/hermes-agent · Image generation: reference image and editing support ## Open now - 3 issues, 0 PRs open on GitHub. - #17034: \[Bug\]: image_edit is registered but not exposed in image_gen/toolsets - #29956: Expose image editing as first-class Hermes tool - #56965: \[Feature\]: Model Native image/video generation Most members concern adding reference image inputs, image-to-image editing, and related tool exposure across image generation backends (Codex, FAL, OpenAI, etc.). One unrelated member addresses CLI multiline paste input handling. ## TL;DR - 15 member(s): 3 open issue(s), 6 closed issue(s), 0 open PR(s), 1 merged PR(s), 5 PR(s) closed without merging. - A fix was merged to main in PR #57017. Verify it addresses your case. - **Recommended:** read PR #57017 (and any competing PRs below) before filing. Confirm the merged change covers your symptom. ## Maintainer resolution verdict - Internal-tracker verdict over this cluster's issue members: `partially_resolved` · deciding PR(s): `pr-57017`. This is distinct from the fix status above. Weigh it yourself. ## How this cluster was grouped - Link evidence (`STRUCTURAL_PARTIAL`): some members are connected by structural evidence (shared changed files, or a closing/superseded reference); the rest are joined by an automated relevance judgement over their text. - Members outside that structural core may belong to a different underlying problem. - Pairwise link density `0.13`: the share of member pairs that carry a link, counting a structural link in full and a relevance-only link at 0.36. `1.00` means every pair is linked. ## Open related issues - #17034: \[Bug\]: image_edit is registered but not exposed in image_gen/toolsets - #29956: Expose image editing as first-class Hermes tool - #56965: \[Feature\]: Model Native image/video generation ## Closed / duplicate issues - #21562: \[Bug\]: native image generation (image_generate) doesn't accept images as input - #21661: openai-codex image provider should support image_edit via Responses image_generation edit - #25661: feat: add reference_image_path support to image_generate tool - #29999: feat: extend image_gen schema to accept reference_image_urls for multi-modal models (e.g. UNI 1.1) - #36737: Image generation tool drops source images for OpenAI/Codex image edits - #42850: \[Feature\]: Support image inputs for capable image generation backends ## Prior & competing PRs - **#21570: fix: forward reference images through Codex and native FAL image generation**. closed without merge · fix tier `suggested` - maintainer direction: category `implemented_elsewhere`; implication `superseded` - _suggested-tier fix detail withheld; re-request with `?tier=all` to include it._ - **#57017: feat(image-gen): support Codex image inputs (salvage #55828)**. merged to main · fix tier `verified` - maintainer direction: category `merged`; implication `superseded` - files: `plugins/image_gen/openai-codex/__init__.py`, `tests/plugins/image_gen/test_openai_codex_provider.py`, `website/docs/reference/tools-reference.md`, `website/docs/user-guide/features/image-generation.md` - fix hunks: - `plugins/image_gen/openai-codex/__init__.py` ```diff @@ -14,19 +14,24 @@ 3. ``image_gen.model`` in ``config.yaml`` (when it's one of our tier IDs) 4. :data:`DEFAULT_MODEL` — ``gpt-image-2-medium`` -Output is saved as PNG under ``$HERMES_HOME/cache/images/``. +Output is saved as PNG under ``$HERMES_HOME/cache/images/``. Source images for +image-to-image/editing are sent as Responses ``input_image`` content parts. """ from __future__ import annotations +import base64 import json import logging +import os +from pathlib import Path from typing import Any, Dict, List, Optional, Tuple from agent.image_gen_provider import ( DEFAULT_ASPECT_RATIO, ImageGenProvider, error_response, + normalize_reference_images, resolve_aspect_ratio, save_b64_image, success_response, ``` ```diff @@ -76,8 +81,18 @@ _CODEX_CHAT_MODEL = "gpt-5.5" _CODEX_BASE_URL = "https://chatgpt.com/backend-api/codex" _CODEX_INSTRUCTIONS = ( - "You are an assistant that must fulfill image generation requests by " - "using the image_generation tool when provided." + "You are an assistant that must fulfill image generation and image editing " + "requests by using the image_generation tool when provided." +) + +_MAX_REFERENCE_IMAGES = 16 +_MAX_INPUT_IMAGE_BYTES = 25 * 1024 * 1024 +# gpt-image-2's Responses ``input_image`` accepts raster formats only. The +# shared magic-byte sniffer also recognizes SVG/TIFF/ICO, which the API +# rejects server-side — gate to this allowlist so unsupported inputs fail +# locally with a clear error instead of an opaque HTTP 400. +_ACCEPTED_INPUT_MIME = frozenset( + {"image/png", "image/jpeg", "image/gif", "image/webp"} ) ``` ```diff @@ -143,8 +158,111 @@ def _read_codex_access_token() -> Optional[str]: return None -def _build_responses_payload(*, prompt: str, size: str, quality: str) -> Dict[str, Any]: +def _sniff_image_mime(raw: bytes) -> Optional[str]: + """Return a safe raster image MIME from magic bytes (not filename labels). + + Delegates magic-byte detection to the shared sniffer in + ``agent.image_routing`` (single source of truth), then gates the result + to :data:`_ACCEPTED_INPUT_MIME` — the raster formats gpt-image-2's + ``input_image`` actually accepts. SVG/TIFF/ICO (which the shared sniffer + also recognizes) are rejected here so they fail locally with a clear + error instead of an opaque server-side HTTP 400. + """ + from agent.image_routing import _sniff_mime_from_bytes + + mime = _sniff_mime_from_bytes(raw) + if mime in _ACCEPTED_INPUT_MIME: + return mime + return None + + +def _data_url_to_input_image_url(value: str) -> str: + """Validate and canonicalize a data:image URL for Responses input_image.""" + if "," not in value: + raise ValueError("Image data URL is missing a comma separator") + header, data = value.split(",", 1) + header_lc = header.lower() + if not header_lc.startswith("data:image/") or ";base64" not in header_lc: + raise ValueError("Only base64 data:image URLs are supported as Codex image inputs") + raw = base64.b64decode(data, validate=True) + if len(raw) > _MAX_INPUT_IMAGE_BYTES: + raise ValueError("Image data URL exceeds 25MB cap") + mime = _sniff_image_mime(raw) + if mime is None: + raise ValueError("Image data URL does not contain supported image bytes") + encoded = base64.b64encode(raw).decode("ascii") + return f"data:{mime};base64,{encoded}" + + +def _local_image_to_data_url(value: str) -> str: + """Read a local image path and return a validated data:image URL.""" + try: + from agent.file_safety import get_read_block_error + + blocked = get_read_block_error(value) + if blocked: + raise ValueError(blocked) + except ValueError: + raise + except Exception as exc: + logger.debug("Codex image input read guard unavailable: %s", exc) + + path = Path(os.path.expanduser(value)).resolve() + if not path.is_file(): + raise ValueError(f"Image input path does not exist or is not a file: {value}") + size = path.stat().st_size + if size <= 0: + raise ValueError(f"Image input path is empty: {value}") + if size > _MAX_INPUT_IMAGE_BYTES: + raise ValueError(f"Image input path exceeds 25MB cap: {value}") + raw = path.read_bytes() + mime = _sniff_image_mime(raw) + if mime is None: + raise ValueError(f"Image input path is not a supported image: {value}") + encoded = base64.b64encode(raw).decode("ascii") + return f"data:{mime};base64,{encoded}" + + +def _to_input_image_part(value: str) -> Dict[str, str]: + """Convert a URL/data URL/local path into a Responses input_image part.""" + candidate = (value or "").strip() + if not candidate: + raise ValueError("Blank image input") + lowered = candidate.lower() + if lowered.startswith("http://") or lowered.startswith("https://"): + image_url = candidate + elif lowered.startswith("data:"): + image_url = _data_url_to_input_image_url(candidate) + else: + image_url = _local_image_to_data_url(candidate) + return {"type": "input_image", "image_url": image_url} + + +def _normalize_input_images( + image_url: Optional[str], + reference_image_urls: Optional[List[str]], +) -> List[Dict[str, str]]: + """Collect primary + reference images as ordered Responses content parts.""" + values: List[str] = [] + if isinstance(image_url, str) and image_url.strip(): + values.append(image_url.strip()) + for ref in (normalize_reference_images(reference_image_urls) or []): + values.append(ref) + values = values[:_MAX_REFERENCE_IMAGES] + return [_to_input_image_part(value) for value in values] + + +def _build_responses_payload( + *, + prompt: str, + size: str, + quality: str, + input_images: Optional[List[Dict[str, str]]] = None, +) -> Dict[str, Any]: """Build the Codex Responses request body for an image_generation call.""" + content: List[Dict[str, Any]] = [{"type": "input_text", "text": prompt}] + if input_images: + content.extend(input_images) return { "model": _CODEX_CHAT_MODEL, "store": False, ``` ```diff @@ -152,7 +270,7 @@ def _build_responses_payload(*, prompt: str, size: str, quality: str) -> Dict[st "input": [{ "type": "message", "role": "user", - "content": [{"type": "input_text", "text": prompt}], + "content": content, }], "tools": [{ "type": "image_generation", ``` ```diff @@ -242,7 +360,14 @@ def flush(): yield payload -def _collect_image_b64(token: str, *, prompt: str, size: str, quality: str) -> Optional[str]: +def _collect_image_b64( + token: str, + *, + prompt: str, + size: str, + quality: str, + input_images: Optional[List[Dict[str, str]]] = None, +) -> Optional[str]: """Stream a Codex Responses image_generation call and return the b64 image.""" import httpx from agent.auxiliary_client import _codex_cloudflare_headers ``` ```diff @@ -253,7 +378,12 @@ def _collect_image_b64(token: str, *, prompt: str, size: str, quality: str) -> O "Authorization": f"Bearer {token}", "Content-Type": "application/json", }) - payload = _build_responses_payload(prompt=prompt, size=size, quality=quality) + payload = _build_responses_payload( + prompt=prompt, + size=size, + quality=quality, + input_images=input_images, + ) timeout = httpx.Timeout(300.0, connect=30.0, read=300.0, write=30.0, pool=30.0) image_b64: Optional[str] = None ``` ```diff @@ -319,7 +449,7 @@ def get_setup_schema(self) -> Dict[str, Any]: return { "name": "OpenAI (Codex auth)", "badge": "free", - "tag": "gpt-image-2 via ChatGPT/Codex OAuth — no API key required (text-to-image only)", + "tag": "gpt-image-2 via ChatGPT/Codex OAuth — no API key required; supports text and image inputs", "env_vars": [], "post_setup_hint": ( "Sign in with `hermes auth codex` (or `hermes setup` → Codex) " ``` ```diff @@ -328,12 +458,11 @@ def get_setup_schema(self) -> Dict[str, Any]: } def capabilities(self) -> Dict[str, Any]: - # The Codex Responses image_generation tool path is text-to-image - # only here. Image-to-image / editing via Codex OAuth is not wired — - # users who need editing should use the `openai` (API key), `fal`, or - # `xai` backends. Declaring text-only keeps the dynamic tool schema - # honest so the model doesn't attempt an unsupported edit. - return {"modalities": ["text"], "max_reference_images": 0} + # The Codex Responses image_generation tool accepts source/reference + # images as `input_image` message content parts. Keep this capability + # honest so the dynamic `image_generate` schema encourages identity- + # preserving edits instead of unrelated text-to-image redraws. + return {"modalities": ["text", "image"], "max_reference_images": _MAX_REFERENCE_IMAGES} def generate( self, ``` ```diff @@ -347,21 +476,6 @@ def generate( prompt = (prompt or "").strip() aspect = resolve_aspect_ratio(aspect_ratio) - # Image-to-image / editing is not supported on the Codex OAuth path. - # Surface a clear, actionable error instead of silently ignoring the - # source image and producing an unrelated picture. - if (isinstance(image_url, str) and image_url.strip()) or reference_image_urls: - return error_response( - error=( - "This model is not capable of image-to-image / editing. " - "Please provide a text-only prompt (drop image_url and " - "reference_image_urls)." - ), - error_type="modality_unsupported", - provider="openai-codex", - aspect_ratio=aspect, - ) - if not prompt: return error_response( error="Prompt is required and must be a non-empty string", ``` ```diff @@ -408,12 +522,25 @@ def generate( aspect_ratio=aspect, ) + try: + input_images = _normalize_input_images(image_url, reference_image_urls) + except Exception as exc: + return error_response( + error=f"Invalid image input for Codex image editing: {exc}", + error_type="invalid_image_input", + provider="openai-codex", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + try: b64 = _collect_image_b64( token, prompt=prompt, size=size, quality=meta["quality"], + input_images=input_images or None, ) except Exception as exc: logger.debug("Codex image generation failed", exc_info=True) ``` ```diff @@ -454,7 +581,8 @@ def generate( prompt=prompt, aspect_ratio=aspect, provider="openai-codex", - extra={"size": size, "quality": meta["quality"]}, + modality="image" if input_images else "text", + extra={"size": size, "quality": meta["quality"], "input_image_count": len(input_images)}, ) ``` - `tests/plugins/image_gen/test_openai_codex_provider.py` ```diff @@ -129,11 +129,12 @@ def test_codex_stream_request_shape(self, provider, monkeypatch): captured = {} - def _collect(token, *, prompt, size, quality): + def _collect(token, *, prompt, size, quality, input_images=None): captured.update(codex_plugin._build_responses_payload( prompt=prompt, size=size, quality=quality, + input_images=input_images, )) return _b64_png() ``` ```diff @@ -160,6 +161,93 @@ def _collect(token, *, prompt, size, quality): assert tool["background"] == "opaque" assert tool["partial_images"] == 1 + def test_capabilities_advertise_image_inputs(self, provider): + caps = provider.capabilities() + assert caps["modalities"] == ["text", "image"] + assert caps["max_reference_images"] == 16 + + def test_codex_stream_request_includes_source_images(self, provider, monkeypatch, tmp_path): + monkeypatch.setattr(codex_plugin, "_read_codex_access_token", lambda: "codex-token") + image_path = tmp_path / "source.png" + image_path.write_bytes(bytes.fromhex(_PNG_HEX)) + + captured = {} + + def _collect(token, *, prompt, size, quality, input_images=None): + captured.update(codex_plugin._build_responses_payload( + prompt=prompt, + size=size, + quality=quality, + input_images=input_images, + )) + return _b64_png() + + monkeypatch.setattr(codex_plugin, "_collect_image_b64", _collect) + + result = provider.generate( + "put this same person in a navy JK uniform", + aspect_ratio="portrait", + image_url=str(image_path), + reference_image_urls=["https://example.com/ref.png"], + ) + + assert result["success"] is True + assert result["modality"] == "image" + assert result["input_image_count"] == 2 + + content = captured["input"][0]["content"] + assert content[0] == { + "type": "input_text", + "text": "put this same person in a navy JK uniform", + } + assert content[1]["type"] == "input_image" + assert content[1]["image_url"].startswith("data:image/png;base64,") + assert content[2] == {"type": "input_image", "image_url": "https://example.com/ref.png"} + + def test_generate_clamps_reference_images_to_cap(self, provider, monkeypatch): + monkeypatch.setattr(codex_plugin, "_read_codex_access_token", lambda: "codex-token") + captured = {} + + def _collect(token, *, prompt, size, quality, input_images=None): + captured["input_images"] = input_images + return _b64_png() + + monkeypatch.setattr(codex_plugin, "_collect_image_b64", _collect) + + refs = [f"https://example.com/ref-{idx}.png" for idx in range(20)] + result = provider.generate("combine the references", reference_image_urls=refs) + + assert result["success"] is True + assert result["modality"] == "image" + assert result["input_image_count"] == 16 + assert len(captured["input_images"]) == 16 + assert captured["input_images"][-1]["image_url"] == "https://example.com/ref-15.png" + + def test_rejects_non_image_local_source(self, provider, monkeypatch, tmp_path): + monkeypatch.setattr(codex_plugin, "_read_codex_access_token", lambda: "codex-token") + text_path = tmp_path / "not-image.txt" + text_path.write_text("hello") + + result = provider.generate("edit this", image_url=str(text_path)) + + assert result["success"] is False + assert result["error_type"] == "invalid_image_input" + assert "not a supported image" in result["error"] + + def test_rejects_svg_local_source(self, provider, monkeypatch, tmp_path): + # The shared magic-byte sniffer recognizes SVG, but gpt-image-2's + # input_image accepts raster only — SVG must fail locally with a clear + # error, not get embedded and rejected server-side with an opaque 400. + monkeypatch.setattr(codex_plugin, "_read_codex_access_token", lambda: "codex-token") + svg_path = tmp_path / "vector.svg" + svg_path.write_text('') + + result = provider.generate("edit this", image_url=str(svg_path)) + + assert result["success"] is False + assert result["error_type"] == "invalid_image_input" + assert "not a supported image" in result["error"] + def test_partial_image_event_used_when_done_missing(self): """If output_item.done is missing, partial_image_b64 is accepted.""" payload = { ``` - `website/docs/reference/tools-reference.md` ```diff @@ -114,7 +114,7 @@ Scoped to the Feishu document-comment handler. Drives comment read/write operati | Tool | Description | Requires environment | |------|-------------|----------------------| -| `image_generate` | Generate images from text prompts (text-to-image) or edit/transform an existing image (image-to-image) via the user-configured backend (FAL.ai, OpenAI, xAI, Krea). Pass `image_url` to edit an image and `reference_image_urls` for style references; omit both for text-to-image. The model is user-configured and not selectable by the agent. Returns a single image URL or local path. | FAL_KEY / OPENAI_API_KEY / xAI OAuth / KREA_API_KEY | +| `image_generate` | Generate images from text prompts (text-to-image) or edit/transform an existing image (image-to-image) via the user-configured backend (FAL.ai, OpenAI, OpenAI Codex auth, xAI, Krea). Pass `image_url` to edit an image and `reference_image_urls` for style references; omit both for text-to-image. The model is user-configured and not selectable by the agent. Returns a single image URL or local path. | FAL_KEY / OPENAI_API_KEY / Codex OAuth / xAI OAuth / KREA_API_KEY | ## `kanban` toolset ``` - `website/docs/user-guide/features/image-generation.md` ```diff @@ -114,7 +114,7 @@ Two inputs drive the edit: | **OpenAI** (`gpt-image-2`) | ✓ | up to 16 | `images.edit()` | | **xAI** (Grok Imagine) | ✓ | 1 | `/v1/images/edits` (`grok-imagine-image-quality`) | | **Krea** (`Krea 2`) | ✓ | up to 10 | reference-guided generation (`image_style_references`) | -| **OpenAI (Codex auth)** | ✗ | — | text-to-image only | +| **OpenAI (Codex auth)** | ✓ | up to 16 | Codex Responses `image_generation` tool with `input_image` content parts | FAL models with an editing endpoint: `flux-2/klein/9b`, `flux-2-pro`, `nano-banana-pro`, `gpt-image-1.5`, `gpt-image-2`, `ideogram/v3`, and ``` - **#19498: fix(cli): defer Enter during rapid text input to prevent multiline paste splitting**. closed without merge - maintainer direction: category `implemented_elsewhere`; implication `superseded`; refs `1de9963ce7`, `#109146` - **#29958: Add image_edit tool for Codex image editing**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `ecffd290a`, `460235d58`, `#55828`, `#57017` - **#49597: feat(image-gen): support Codex image editing**. closed without merge - maintainer direction: category `wontfix_bydesign`; implication `wontfix`; refs `d5aba469886f`, `8cb12d883422`, `f9d9ce5652a9` - **#55828: feat(image-gen): support Codex image inputs**. closed without merge - maintainer direction: category `merged_via`; implication `superseded`; refs `ecffd290a`, `460235d58`, `#57017` ## Fix status - Merged to main in PR #57017 (cited above). No release data is tracked. Verify downstream. - Fix last verified (merged) `2026-07-02`. - Subsystem: `agent`