Vision — Local Image Querying via Ollama

Ask natural-language questions about images without passing them to the main agent as visual input. Useful for verifying screenshots, annotating assets, or building automated checks around visual output.

When to Use This Skill

• Describing a screenshot for a PR description or user-facing document
• Checking whether an automated browser run produced visible canvas content
• Asking "do any elements overlap?" on a rendered output
• Any question where the answer is in the pixels but you don't want to use vision tokens in the main context

---

Quick Reference

All commands use uv run — dependencies are installed automatically.

SCRIPT=.agents/skills/vision/scripts/ask.py

# health check (fast, no image, confirms Ollama + model respond)
uv run $SCRIPT --ping

# system info — memory, storage, installed models
uv run $SCRIPT --info
uv run $SCRIPT --memory
uv run $SCRIPT --storage

# describe an image (default prompt)
uv run $SCRIPT path/to/image.png

# explicit shortcut
uv run $SCRIPT path/to/image.png describe

# custom question
uv run $SCRIPT path/to/image.png \
  --prompt "Do you see any overlapping UI elements?"

uv run $SCRIPT canvas.png \
  --prompt "Does this canvas contain any designed content, or is it empty?"

# optional: pin a specific Gemma 4 tag (default is any installed gemma4)
uv run $SCRIPT image.png --model gemma4:e4b

# list installed Gemma 4 vision models
uv run $SCRIPT --list-models

---

Prerequisites

Ollama must be running locally. The script connects to http://localhost:11434 and fails immediately if it cannot reach it.

# start Ollama (if not already running)
ollama serve

# install Gemma 4 (multimodal — required for this skill)
ollama pull gemma4

The script does not install models. If Gemma 4 is not installed it prints the list of installed models and a pull suggestion, then exits.

uv is required to run the script (handles dependency installation automatically). No requirements.txt or manual pip install needed.

---

Model Selection

This skill uses only Gemma 4 on Ollama (gemma4 and tags such as gemma4:latest, gemma4:e4b). Other multimodal models are ignored so agents do not silently fall back to a different family.

When --model is omitted, the script picks any installed gemma4 tag (for example gemma4:latest). Use --model gemma4:e4b (or another tag) to pin a specific variant.

---

System Info

Before running a heavy query, check whether the machine has enough resources. This is optional — the script does not enforce limits — but useful context for deciding whether to proceed or skip.

uv run $SCRIPT --info      # memory + storage + model list
uv run $SCRIPT --memory    # just memory
uv run $SCRIPT --storage   # just storage

Tip: on machines with ≤8 GB RAM, large vision models may cause swapping or OOM. Consider a smaller Gemma 4 variant (for example gemma4:e2b) or skip the query.

---

Behavior

• Fails fast if Ollama is unreachable or Gemma 4 is not installed. Exit code is non-zero; the error message includes a hint or pull command.
• Sequential only — Ollama is a single-worker process. Never call ask.py in parallel (e.g. two concurrent tool calls). Queue calls one at a time.
• No side effects beyond the local Ollama process.
• Auto-installs deps via uv inline script metadata (PEP 723). Only dependency is the ollama Python package.
• Supported formats: .png, .jpg, .jpeg, .webp, .gif, .bmp.

---

Typical Agent Workflow

1. A tool (browser automation, screenshot capture, golden renderer) writes an image to disk.
2. Call ask.py with a targeted prompt suited to the task.
3. Parse the text response to decide the next action.

# Quick sanity check first
uv run $SCRIPT --ping

# Verify a browser screenshot has content before including it in a doc
uv run $SCRIPT /tmp/preview.png \
  --prompt "Answer with YES or NO: does this screenshot show any visible UI content, shapes, or text?"

# Describe a captured screenshot for a PR description
uv run $SCRIPT /tmp/canvas-screenshot.png \
  --prompt "Describe what visual effect is shown. Be specific about blur, colors, and shapes."

---

Troubleshooting

Symptom	Cause	Fix
cannot reach Ollama	Ollama not running	ollama serve
no Gemma 4 vision model found	Gemma 4 not installed	ollama pull gemma4
model 'X' is not available	Model name typo or not installed	--list-models to see what's installed
Slow response	Large model on CPU	Try a smaller tag (e.g. gemma4:e2b)
Vague or wrong answer	Generic prompt	Write a more specific --prompt
'ollama' package not found	Not using uv run	Run with uv run ask.py instead