{"repo":"NousResearch/hermes-agent","public_id":"17034","display_label":"Image generation: reference image and editing support","summary":"Most members concern adding reference image inputs, image-to-image editing, and related tool exposure across image generation backends (Codex, FAL, OpenAI, etc.). One unrelated member addresses CLI multiline paste input handling.","subsystem":"agent","counts":{"members":15,"open_issues":3,"open_issues_distinct":3,"open_prs":0,"closed_issues":6,"merged_prs":1,"attempt_prs":5},"landed_fix":{"status":"merged_to_main","basis":"structural","pr":57017},"resolution":{"state":"partially_resolved","by":"pr-57017"},"freshness":{"state":"verified","last_verified_at":"2026-07-02T11:42:25+00:00","is_stale":false},"grouping":{"backbone":"STRUCTURAL_PARTIAL","link_density":0.13371428571428565},"fix_edges":[{"issue_number":21661,"pr_number":57017,"tier":"verified","basis":"closing_ref","confidence":0.95,"recoverable_from_timeline":true},{"issue_number":29956,"pr_number":57017,"tier":"verified","basis":"superseded_ref","confidence":0.9,"recoverable_from_timeline":false},{"issue_number":36737,"pr_number":57017,"tier":"verified","basis":"closing_ref","confidence":0.95,"recoverable_from_timeline":true}],"members":[{"kind":"issue","number":17034,"title":"[Bug]: image_edit is registered but not exposed in image_gen/toolsets","url":null,"state":"open"},{"kind":"issue","number":21562,"title":"[Bug]: native image generation (image_generate) doesn't accept images as input","url":null,"state":"closed"},{"kind":"issue","number":21661,"title":"openai-codex image provider should support image_edit via Responses image_generation edit","url":null,"state":"closed"},{"kind":"issue","number":25661,"title":"feat: add reference_image_path support to image_generate tool","url":null,"state":"closed"},{"kind":"issue","number":29956,"title":"Expose image editing as first-class Hermes tool","url":null,"state":"open"},{"kind":"issue","number":29999,"title":"feat: extend image_gen schema to accept reference_image_urls for multi-modal models (e.g. UNI 1.1)","url":null,"state":"closed"},{"kind":"issue","number":36737,"title":"Image generation tool drops source images for OpenAI/Codex image edits","url":null,"state":"closed"},{"kind":"issue","number":42850,"title":"[Feature]: Support image inputs for capable image generation backends","url":null,"state":"closed"},{"kind":"issue","number":56965,"title":"[Feature]: Model Native image/video generation","url":null,"state":"open"},{"kind":"pr","number":19498,"title":"fix(cli): defer Enter during rapid text input to prevent multiline paste splitting","url":null,"state":"closed","close_reason":{"refs":[{"kind":"commit","value":"1de9963ce7","number":0},{"kind":"pr","value":"#109146","number":109146}],"category":"implemented_elsewhere","implication":"superseded"},"merged":false},{"kind":"pr","number":21570,"title":"fix: forward reference images through Codex and native FAL image generation","url":null,"state":"closed","close_reason":{"category":"implemented_elsewhere","implication":"superseded"},"merged":false,"is_fix_pr":true,"fix_tier":"suggested","fix_detail_withheld":true},{"kind":"pr","number":29958,"title":"Add image_edit tool for Codex image editing","url":null,"state":"closed","close_reason":{"refs":[{"kind":"commit","value":"ecffd290a","number":0},{"kind":"commit","value":"460235d58","number":0},{"kind":"pr","value":"#55828","number":55828},{"kind":"pr","value":"#57017","number":57017}],"category":"merged_via","implication":"superseded"},"merged":false},{"kind":"pr","number":49597,"title":"feat(image-gen): support Codex image editing","url":null,"state":"closed","close_reason":{"refs":[{"kind":"commit","value":"d5aba469886f","number":0},{"kind":"commit","value":"8cb12d883422","number":0},{"kind":"commit","value":"f9d9ce5652a9","number":0}],"category":"wontfix_bydesign","implication":"wontfix"},"merged":false},{"kind":"pr","number":55828,"title":"feat(image-gen): support Codex image inputs","url":null,"state":"closed","close_reason":{"refs":[{"kind":"commit","value":"ecffd290a","number":0},{"kind":"commit","value":"460235d58","number":0},{"kind":"pr","value":"#57017","number":57017}],"category":"merged_via","implication":"superseded"},"merged":false},{"kind":"pr","number":57017,"title":"feat(image-gen): support Codex image inputs (salvage #55828)","url":null,"state":"merged","close_reason":{"category":"merged","implication":"superseded"},"merged":true,"is_fix_pr":true,"fix_tier":"verified","files_changed":["plugins/image_gen/openai-codex/__init__.py","tests/plugins/image_gen/test_openai_codex_provider.py","website/docs/reference/tools-reference.md","website/docs/user-guide/features/image-generation.md"],"fix_hunks":[{"file":"plugins/image_gen/openai-codex/__init__.py","hunks":["@@ -14,19 +14,24 @@\n 3. ``image_gen.model`` in ``config.yaml`` (when it's one of our tier IDs)\n 4. :data:`DEFAULT_MODEL` — ``gpt-image-2-medium``\n \n-Output is saved as PNG under ``$HERMES_HOME/cache/images/``.\n+Output is saved as PNG under ``$HERMES_HOME/cache/images/``. Source images for\n+image-to-image/editing are sent as Responses ``input_image`` content parts.\n \"\"\"\n \n from __future__ import annotations\n \n+import base64\n import json\n import logging\n+import os\n+from pathlib import Path\n from typing import Any, Dict, List, Optional, Tuple\n \n from agent.image_gen_provider import (\n     DEFAULT_ASPECT_RATIO,\n     ImageGenProvider,\n     error_response,\n+    normalize_reference_images,\n     resolve_aspect_ratio,\n     save_b64_image,\n     success_response,","@@ -76,8 +81,18 @@\n _CODEX_CHAT_MODEL = \"gpt-5.5\"\n _CODEX_BASE_URL = \"https://chatgpt.com/backend-api/codex\"\n _CODEX_INSTRUCTIONS = (\n-    \"You are an assistant that must fulfill image generation requests by \"\n-    \"using the image_generation tool when provided.\"\n+    \"You are an assistant that must fulfill image generation and image editing \"\n+    \"requests by using the image_generation tool when provided.\"\n+)\n+\n+_MAX_REFERENCE_IMAGES = 16\n+_MAX_INPUT_IMAGE_BYTES = 25 * 1024 * 1024\n+# gpt-image-2's Responses ``input_image`` accepts raster formats only. The\n+# shared magic-byte sniffer also recognizes SVG/TIFF/ICO, which the API\n+# rejects server-side — gate to this allowlist so unsupported inputs fail\n+# locally with a clear error instead of an opaque HTTP 400.\n+_ACCEPTED_INPUT_MIME = frozenset(\n+    {\"image/png\", \"image/jpeg\", \"image/gif\", \"image/webp\"}\n )\n \n ","@@ -143,8 +158,111 @@ def _read_codex_access_token() -> Optional[str]:\n         return None\n \n \n-def _build_responses_payload(*, prompt: str, size: str, quality: str) -> Dict[str, Any]:\n+def _sniff_image_mime(raw: bytes) -> Optional[str]:\n+    \"\"\"Return a safe raster image MIME from magic bytes (not filename labels).\n+\n+    Delegates magic-byte detection to the shared sniffer in\n+    ``agent.image_routing`` (single source of truth), then gates the result\n+    to :data:`_ACCEPTED_INPUT_MIME` — the raster formats gpt-image-2's\n+    ``input_image`` actually accepts. SVG/TIFF/ICO (which the shared sniffer\n+    also recognizes) are rejected here so they fail locally with a clear\n+    error instead of an opaque server-side HTTP 400.\n+    \"\"\"\n+    from agent.image_routing import _sniff_mime_from_bytes\n+\n+    mime = _sniff_mime_from_bytes(raw)\n+    if mime in _ACCEPTED_INPUT_MIME:\n+        return mime\n+    return None\n+\n+\n+def _data_url_to_input_image_url(value: str) -> str:\n+    \"\"\"Validate and canonicalize a data:image URL for Responses input_image.\"\"\"\n+    if \",\" not in value:\n+        raise ValueError(\"Image data URL is missing a comma separator\")\n+    header, data = value.split(\",\", 1)\n+    header_lc = header.lower()\n+    if not header_lc.startswith(\"data:image/\") or \";base64\" not in header_lc:\n+        raise ValueError(\"Only base64 data:image URLs are supported as Codex image inputs\")\n+    raw = base64.b64decode(data, validate=True)\n+    if len(raw) > _MAX_INPUT_IMAGE_BYTES:\n+        raise ValueError(\"Image data URL exceeds 25MB cap\")\n+    mime = _sniff_image_mime(raw)\n+    if mime is None:\n+        raise ValueError(\"Image data URL does not contain supported image bytes\")\n+    encoded = base64.b64encode(raw).decode(\"ascii\")\n+    return f\"data:{mime};base64,{encoded}\"\n+\n+\n+def _local_image_to_data_url(value: str) -> str:\n+    \"\"\"Read a local image path and return a validated data:image URL.\"\"\"\n+    try:\n+        from agent.file_safety import get_read_block_error\n+\n+        blocked = get_read_block_error(value)\n+        if blocked:\n+            raise ValueError(blocked)\n+    except ValueError:\n+        raise\n+    except Exception as exc:\n+        logger.debug(\"Codex image input read guard unavailable: %s\", exc)\n+\n+    path = Path(os.path.expanduser(value)).resolve()\n+    if not path.is_file():\n+        raise ValueError(f\"Image input path does not exist or is not a file: {value}\")\n+    size = path.stat().st_size\n+    if size <= 0:\n+        raise ValueError(f\"Image input path is empty: {value}\")\n+    if size > _MAX_INPUT_IMAGE_BYTES:\n+        raise ValueError(f\"Image input path exceeds 25MB cap: {value}\")\n+    raw = path.read_bytes()\n+    mime = _sniff_image_mime(raw)\n+    if mime is None:\n+        raise ValueError(f\"Image input path is not a supported image: {value}\")\n+    encoded = base64.b64encode(raw).decode(\"ascii\")\n+    return f\"data:{mime};base64,{encoded}\"\n+\n+\n+def _to_input_image_part(value: str) -> Dict[str, str]:\n+    \"\"\"Convert a URL/data URL/local path into a Responses input_image part.\"\"\"\n+    candidate = (value or \"\").strip()\n+    if not candidate:\n+        raise ValueError(\"Blank image input\")\n+    lowered = candidate.lower()\n+    if lowered.startswith(\"http://\") or lowered.startswith(\"https://\"):\n+        image_url = candidate\n+    elif lowered.startswith(\"data:\"):\n+        image_url = _data_url_to_input_image_url(candidate)\n+    else:\n+        image_url = _local_image_to_data_url(candidate)\n+    return {\"type\": \"input_image\", \"image_url\": image_url}\n+\n+\n+def _normalize_input_images(\n+    image_url: Optional[str],\n+    reference_image_urls: Optional[List[str]],\n+) -> List[Dict[str, str]]:\n+    \"\"\"Collect primary + reference images as ordered Responses content parts.\"\"\"\n+    values: List[str] = []\n+    if isinstance(image_url, str) and image_url.strip():\n+        values.append(image_url.strip())\n+    for ref in (normalize_reference_images(reference_image_urls) or []):\n+        values.append(ref)\n+    values = values[:_MAX_REFERENCE_IMAGES]\n+    return [_to_input_image_part(value) for value in values]\n+\n+\n+def _build_responses_payload(\n+    *,\n+    prompt: str,\n+    size: str,\n+    quality: str,\n+    input_images: Optional[List[Dict[str, str]]] = None,\n+) -> Dict[str, Any]:\n     \"\"\"Build the Codex Responses request body for an image_generation call.\"\"\"\n+    content: List[Dict[str, Any]] = [{\"type\": \"input_text\", \"text\": prompt}]\n+    if input_images:\n+        content.extend(input_images)\n     return {\n         \"model\": _CODEX_CHAT_MODEL,\n         \"store\": False,","@@ -152,7 +270,7 @@ def _build_responses_payload(*, prompt: str, size: str, quality: str) -> Dict[st\n         \"input\": [{\n             \"type\": \"message\",\n             \"role\": \"user\",\n-            \"content\": [{\"type\": \"input_text\", \"text\": prompt}],\n+            \"content\": content,\n         }],\n         \"tools\": [{\n             \"type\": \"image_generation\",","@@ -242,7 +360,14 @@ def flush():\n         yield payload\n \n \n-def _collect_image_b64(token: str, *, prompt: str, size: str, quality: str) -> Optional[str]:\n+def _collect_image_b64(\n+    token: str,\n+    *,\n+    prompt: str,\n+    size: str,\n+    quality: str,\n+    input_images: Optional[List[Dict[str, str]]] = None,\n+) -> Optional[str]:\n     \"\"\"Stream a Codex Responses image_generation call and return the b64 image.\"\"\"\n     import httpx\n     from agent.auxiliary_client import _codex_cloudflare_headers","@@ -253,7 +378,12 @@ def _collect_image_b64(token: str, *, prompt: str, size: str, quality: str) -> O\n         \"Authorization\": f\"Bearer {token}\",\n         \"Content-Type\": \"application/json\",\n     })\n-    payload = _build_responses_payload(prompt=prompt, size=size, quality=quality)\n+    payload = _build_responses_payload(\n+        prompt=prompt,\n+        size=size,\n+        quality=quality,\n+        input_images=input_images,\n+    )\n     timeout = httpx.Timeout(300.0, connect=30.0, read=300.0, write=30.0, pool=30.0)\n \n     image_b64: Optional[str] = None","@@ -319,7 +449,7 @@ def get_setup_schema(self) -> Dict[str, Any]:\n         return {\n             \"name\": \"OpenAI (Codex auth)\",\n             \"badge\": \"free\",\n-            \"tag\": \"gpt-image-2 via ChatGPT/Codex OAuth — no API key required (text-to-image only)\",\n+            \"tag\": \"gpt-image-2 via ChatGPT/Codex OAuth — no API key required; supports text and image inputs\",\n             \"env_vars\": [],\n             \"post_setup_hint\": (\n                 \"Sign in with `hermes auth codex` (or `hermes setup` → Codex) \"","@@ -328,12 +458,11 @@ def get_setup_schema(self) -> Dict[str, Any]:\n         }\n \n     def capabilities(self) -> Dict[str, Any]:\n-        # The Codex Responses image_generation tool path is text-to-image\n-        # only here. Image-to-image / editing via Codex OAuth is not wired —\n-        # users who need editing should use the `openai` (API key), `fal`, or\n-        # `xai` backends. Declaring text-only keeps the dynamic tool schema\n-        # honest so the model doesn't attempt an unsupported edit.\n-        return {\"modalities\": [\"text\"], \"max_reference_images\": 0}\n+        # The Codex Responses image_generation tool accepts source/reference\n+        # images as `input_image` message content parts. Keep this capability\n+        # honest so the dynamic `image_generate` schema encourages identity-\n+        # preserving edits instead of unrelated text-to-image redraws.\n+        return {\"modalities\": [\"text\", \"image\"], \"max_reference_images\": _MAX_REFERENCE_IMAGES}\n \n     def generate(\n         self,","@@ -347,21 +476,6 @@ def generate(\n         prompt = (prompt or \"\").strip()\n         aspect = resolve_aspect_ratio(aspect_ratio)\n \n-        # Image-to-image / editing is not supported on the Codex OAuth path.\n-        # Surface a clear, actionable error instead of silently ignoring the\n-        # source image and producing an unrelated picture.\n-        if (isinstance(image_url, str) and image_url.strip()) or reference_image_urls:\n-            return error_response(\n-                error=(\n-                    \"This model is not capable of image-to-image / editing. \"\n-                    \"Please provide a text-only prompt (drop image_url and \"\n-                    \"reference_image_urls).\"\n-                ),\n-                error_type=\"modality_unsupported\",\n-                provider=\"openai-codex\",\n-                aspect_ratio=aspect,\n-            )\n-\n         if not prompt:\n             return error_response(\n                 error=\"Prompt is required and must be a non-empty string\",","@@ -408,12 +522,25 @@ def generate(\n                 aspect_ratio=aspect,\n             )\n \n+        try:\n+            input_images = _normalize_input_images(image_url, reference_image_urls)\n+        except Exception as exc:\n+            return error_response(\n+                error=f\"Invalid image input for Codex image editing: {exc}\",\n+                error_type=\"invalid_image_input\",\n+                provider=\"openai-codex\",\n+                model=tier_id,\n+                prompt=prompt,\n+                aspect_ratio=aspect,\n+            )\n+\n         try:\n             b64 = _collect_image_b64(\n                 token,\n                 prompt=prompt,\n                 size=size,\n                 quality=meta[\"quality\"],\n+                input_images=input_images or None,\n             )\n         except Exception as exc:\n             logger.debug(\"Codex image generation failed\", exc_info=True)","@@ -454,7 +581,8 @@ def generate(\n             prompt=prompt,\n             aspect_ratio=aspect,\n             provider=\"openai-codex\",\n-            extra={\"size\": size, \"quality\": meta[\"quality\"]},\n+            modality=\"image\" if input_images else \"text\",\n+            extra={\"size\": size, \"quality\": meta[\"quality\"], \"input_image_count\": len(input_images)},\n         )\n \n "]},{"file":"tests/plugins/image_gen/test_openai_codex_provider.py","hunks":["@@ -129,11 +129,12 @@ def test_codex_stream_request_shape(self, provider, monkeypatch):\n \n         captured = {}\n \n-        def _collect(token, *, prompt, size, quality):\n+        def _collect(token, *, prompt, size, quality, input_images=None):\n             captured.update(codex_plugin._build_responses_payload(\n                 prompt=prompt,\n                 size=size,\n                 quality=quality,\n+                input_images=input_images,\n             ))\n             return _b64_png()\n ","@@ -160,6 +161,93 @@ def _collect(token, *, prompt, size, quality):\n         assert tool[\"background\"] == \"opaque\"\n         assert tool[\"partial_images\"] == 1\n \n+    def test_capabilities_advertise_image_inputs(self, provider):\n+        caps = provider.capabilities()\n+        assert caps[\"modalities\"] == [\"text\", \"image\"]\n+        assert caps[\"max_reference_images\"] == 16\n+\n+    def test_codex_stream_request_includes_source_images(self, provider, monkeypatch, tmp_path):\n+        monkeypatch.setattr(codex_plugin, \"_read_codex_access_token\", lambda: \"codex-token\")\n+        image_path = tmp_path / \"source.png\"\n+        image_path.write_bytes(bytes.fromhex(_PNG_HEX))\n+\n+        captured = {}\n+\n+        def _collect(token, *, prompt, size, quality, input_images=None):\n+            captured.update(codex_plugin._build_responses_payload(\n+                prompt=prompt,\n+                size=size,\n+                quality=quality,\n+                input_images=input_images,\n+            ))\n+            return _b64_png()\n+\n+        monkeypatch.setattr(codex_plugin, \"_collect_image_b64\", _collect)\n+\n+        result = provider.generate(\n+            \"put this same person in a navy JK uniform\",\n+            aspect_ratio=\"portrait\",\n+            image_url=str(image_path),\n+            reference_image_urls=[\"https://example.com/ref.png\"],\n+        )\n+\n+        assert result[\"success\"] is True\n+        assert result[\"modality\"] == \"image\"\n+        assert result[\"input_image_count\"] == 2\n+\n+        content = captured[\"input\"][0][\"content\"]\n+        assert content[0] == {\n+            \"type\": \"input_text\",\n+            \"text\": \"put this same person in a navy JK uniform\",\n+        }\n+        assert content[1][\"type\"] == \"input_image\"\n+        assert content[1][\"image_url\"].startswith(\"data:image/png;base64,\")\n+        assert content[2] == {\"type\": \"input_image\", \"image_url\": \"https://example.com/ref.png\"}\n+\n+    def test_generate_clamps_reference_images_to_cap(self, provider, monkeypatch):\n+        monkeypatch.setattr(codex_plugin, \"_read_codex_access_token\", lambda: \"codex-token\")\n+        captured = {}\n+\n+        def _collect(token, *, prompt, size, quality, input_images=None):\n+            captured[\"input_images\"] = input_images\n+            return _b64_png()\n+\n+        monkeypatch.setattr(codex_plugin, \"_collect_image_b64\", _collect)\n+\n+        refs = [f\"https://example.com/ref-{idx}.png\" for idx in range(20)]\n+        result = provider.generate(\"combine the references\", reference_image_urls=refs)\n+\n+        assert result[\"success\"] is True\n+        assert result[\"modality\"] == \"image\"\n+        assert result[\"input_image_count\"] == 16\n+        assert len(captured[\"input_images\"]) == 16\n+        assert captured[\"input_images\"][-1][\"image_url\"] == \"https://example.com/ref-15.png\"\n+\n+    def test_rejects_non_image_local_source(self, provider, monkeypatch, tmp_path):\n+        monkeypatch.setattr(codex_plugin, \"_read_codex_access_token\", lambda: \"codex-token\")\n+        text_path = tmp_path / \"not-image.txt\"\n+        text_path.write_text(\"hello\")\n+\n+        result = provider.generate(\"edit this\", image_url=str(text_path))\n+\n+        assert result[\"success\"] is False\n+        assert result[\"error_type\"] == \"invalid_image_input\"\n+        assert \"not a supported image\" in result[\"error\"]\n+\n+    def test_rejects_svg_local_source(self, provider, monkeypatch, tmp_path):\n+        # The shared magic-byte sniffer recognizes SVG, but gpt-image-2's\n+        # input_image accepts raster only — SVG must fail locally with a clear\n+        # error, not get embedded and rejected server-side with an opaque 400.\n+        monkeypatch.setattr(codex_plugin, \"_read_codex_access_token\", lambda: \"codex-token\")\n+        svg_path = tmp_path / \"vector.svg\"\n+        svg_path.write_text('<svg xmlns=\"http://www.w3.org/2000/svg\"></svg>')\n+\n+        result = provider.generate(\"edit this\", image_url=str(svg_path))\n+\n+        assert result[\"success\"] is False\n+        assert result[\"error_type\"] == \"invalid_image_input\"\n+        assert \"not a supported image\" in result[\"error\"]\n+\n     def test_partial_image_event_used_when_done_missing(self):\n         \"\"\"If output_item.done is missing, partial_image_b64 is accepted.\"\"\"\n         payload = {"]},{"file":"website/docs/reference/tools-reference.md","hunks":["@@ -114,7 +114,7 @@ Scoped to the Feishu document-comment handler. Drives comment read/write operati\n \n | Tool | Description | Requires environment |\n |------|-------------|----------------------|\n-| `image_generate` | Generate images from text prompts (text-to-image) or edit/transform an existing image (image-to-image) via the user-configured backend (FAL.ai, OpenAI, xAI, Krea). Pass `image_url` to edit an image and `reference_image_urls` for style references; omit both for text-to-image. The model is user-configured and not selectable by the agent. Returns a single image URL or local path. | FAL_KEY / OPENAI_API_KEY / xAI OAuth / KREA_API_KEY |\n+| `image_generate` | Generate images from text prompts (text-to-image) or edit/transform an existing image (image-to-image) via the user-configured backend (FAL.ai, OpenAI, OpenAI Codex auth, xAI, Krea). Pass `image_url` to edit an image and `reference_image_urls` for style references; omit both for text-to-image. The model is user-configured and not selectable by the agent. Returns a single image URL or local path. | FAL_KEY / OPENAI_API_KEY / Codex OAuth / xAI OAuth / KREA_API_KEY |\n \n ## `kanban` toolset\n "]},{"file":"website/docs/user-guide/features/image-generation.md","hunks":["@@ -114,7 +114,7 @@ Two inputs drive the edit:\n | **OpenAI** (`gpt-image-2`) | ✓ | up to 16 | `images.edit()` |\n | **xAI** (Grok Imagine) | ✓ | 1 | `/v1/images/edits` (`grok-imagine-image-quality`) |\n | **Krea** (`Krea 2`) | ✓ | up to 10 | reference-guided generation (`image_style_references`) |\n-| **OpenAI (Codex auth)** | ✗ | — | text-to-image only |\n+| **OpenAI (Codex auth)** | ✓ | up to 16 | Codex Responses `image_generation` tool with `input_image` content parts |\n \n FAL models with an editing endpoint: `flux-2/klein/9b`, `flux-2-pro`,\n `nano-banana-pro`, `gpt-image-1.5`, `gpt-image-2`, `ideogram/v3`, and\n"]}]}],"trust":{"default_tier":"verified","includes_suggested":false}}