gemini-er

API-backed 2D detection on Gemini Robotics-ER. Zero GPU. The canonical perception recipe (from the dev tree's perceive_gemini_er workflow script) is: gemini-er.detect → best box by score → sam3.segment_box on the full frame at that box → depth projection → OBB fit.

Install

uv sync --extra gemini-er    # google-genai  (pip: pip install -e ".[gemini-er]")
export GOOGLE_API_KEY=...    # or GEMINI_API_KEY

Config

Env	Meaning	Default
GAP_GEMINI_ER_MODEL	Gemini model name	gemini-robotics-er-1.5-preview
GOOGLE_API_KEY / GEMINI_API_KEY	API key (SDK default resolution)	—

Contract

gemini-er.detect(image, query) returns {"detections": [{"box": BoundingBox2D, "label": str, "score": float}]}:

• box is pixel-space {x1, y1, x2, y2} (top-left → bottom-right), clamped to the image bounds. The model emits the Gemini box_2d convention ([ymin, xmin, ymax, xmax] normalized 0–1000); conversion happens here.
• score defaults to 1.0 when the model reports none — callers select the best detection with max(..., key=score).
• No match (or unparseable model output) → empty detections, never an error. Treat empty as "object not visible".

When to use

• Detection boxes for open-vocabulary prompts, no local weights.
• Prefer molmo.point_prompt when a single click point is enough, and vlm.query_yes_no for semantic checks without localization.