Local Model Selection for Hermes

Use this skill when the user asks which local model to run, which quant to pick, which uncensored variant to use, how to pair models across GPUs, or what auxiliary model to use alongside their main model.

When to use

Trigger on requests like:

• "what's the best Qwen model for my hardware"
• "which quant should I use for 32GB VRAM"
• "what uncensored model for 16GB"
• "what should I run as a secondary/auxiliary model"
• "compare these three models for agent use"
• "is this model still the best or has something newer dropped"
• "where are my LM Studio models"
• "move/delete these LM Studio models to free space"
• "are there any Ollama models on this Mac"
• "uninstall Ollama"

For storage operations, distinguish the machine being cleaned from any remote model host. For LM Studio, inventory exact model directories, check lms ps before deletion, delete only explicitly named model repositories, and verify both path removal and real free-space change; see references/lm-studio-model-storage-operations.md. For Ollama on macOS, cross-check ollama list, allocated model blobs/manifests, and large files before claiming models exist, then use the process/app/CLI/package/data verification sequence in references/macos-ollama-storage-and-uninstall.md for a complete uninstall.

Core rule: verify model existence before recommending

Never fabricate model names. Model families have specific size tiers — not every number exists in every family. Before recommending, verify the model exists on HuggingFace or Ollama.

When a user asks for recommendations:

• Use exact model identifiers, e.g. "Qwen3.6-27B" not just "a Qwen 3 model."
• Never say "we can run about X parameters" without naming at least one concrete, released, GGUF-available candidate that actually fits.
• If you're unsure of the latest family/size, check via web_search or HuggingFace before answering — the user will spot vague or outdated claims immediately.

If you mix up versions (e.g., saying "Qwen 3.5" when Qwen 3.6 is current), correct yourself directly instead of hedging.

Model family naming conventions

Qwen family

Family	Sizes	Key dates
Qwen3	0.6B, 1.7B, 4B, 8B, 14B, 32B	Dec 2025
Qwen3.5	0.8B, 2B, 4B, 9B, 27B, 35B-A3B, 122B-A10B	Feb–Jul 2026
Qwen3.6	27B, 35B-A3B, 40B (Claude distill)	Apr 2026
Qwen-AgentWorld	35B-A3B, 397B-A17B	Jun 2026 — world-model CPT/SFT/RL atop Qwen3.5 MoE, 256K ctx

Common pitfall: No Qwen3.5 14B exists. Qwen3 has 14B; Qwen3.5 has 9B. Do not combine family names with wrong sizes.

AgentWorld positioning: Qwen-AgentWorld-35B-A3B is best treated as a world-model / agent-worker candidate, not a drop-in replacement for Qwen3.6-27B dense as the main Hermes brain. Early community reports are promising for long, non-coding agent tasks and local tool-heavy workflows, but mixed for coding. Before judging quality, verify the chat template/tool-call formatting; a patched Qwen chat_template.jinja may be necessary for reliable role/tool behavior.

Gemma 4 family (Apr 2026)

Size	Type	Notes
12B	Dense	Vision + tool calling
26B-A4B	MoE (~4B active)	Vision + tool calling, good 16GB pick
31B	Dense	Vision + tool calling

Other relevant families

• GPT-OSS 20B — OpenAI MoE (20B total, 4-6 active), 128K ctx
• Mistral Small 24B — Dense, Apache 2.0, naturally less restricted base training
• Dolphin 3.0 — Dataset-filtered uncensored variants (Mistral 24B, Llama 8B)

Uncensored/abliterated landscape

Three main techniques:

Method	How	Trade-off
Heretic	LoRA-based refusal vector extraction	Lowest KL divergence, best capability preservation
HauhauCS	Aggressive/Balanced variants	Balanced variant tuned for agentic stability. Aggressive strips preamble.
Huihui	Crude abliteration (proof-of-concept)	Can cause catastrophic degradation at larger scales (KL >3 on 4B+)

Key findings from community benchmarks (Nathan Sapwell, Apr 2026)

• Abliteration is NOT lossless at any scale. All techniques cause measurable benchmark drops.
• Base Qwen3.6-27B refuses 99.5% of harmful prompts. Heretic and HauhauCS both reduce to near-zero while preserving most benchmarks.
• Huihui degrades catastrophically on models >4B — avoid for serious agent work.
• HauhauCS Balanced is specifically recommended for agentic coding — keeps self-reasoning preamble which stabilizes long tool-call chains.

Where to find uncensored GGUFs

• DavidAU (HuggingFace) — Heretic + NEO-CODE-Di-IMatrix quants for many models
• mradermacher (HuggingFace) — static quants of Heretic variants (Gemma 4 especially)
• bartowski (HuggingFace) — standard quants of most models including abliterated
• huihui-ai (HuggingFace + Ollama) — crude abliterated variants
• HauhauCS (HuggingFace) — Aggressive and Balanced uncensored variants

VRAM tier recommendations (as of June 2026)

16GB VRAM tier (RTX 5070 Ti-class cards)

Rank	Model	Repo	Quant	VRAM	Notes
1	Gemma 4 26B-A4B Heretic	mradermacher/gemma-4-26B-A4B-it-heretic-GGUF	Q4_K_M	~14GB	MoE, vision, tool calling. InsiderLLM top pick.
2	Mistral Small 24B Abliterated	various on HF	Q4_K_M	~14GB	Dense, BSWEN #1 pick, limited ctx room
3	GPT-OSS 20B Heretic	DavidAU/OpenAi-GPT-oss-20b-HERETIC-uncensored-NEO-Imatrix-gguf	IQ4_NL	~12GB	Highest benchmark scores, MoE can be quirky
4	Qwen3 14B Abliterated	huihui-ai/Qwen3-14B-abliterated	Q4_K_M	~10.7GB	Same Qwen DNA as main 27B, lots of headroom
5	Qwen3.5 9B Uncensored	HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive	Q4_K_M	~6GB	Lightning fast, weak for complex aux tasks

24GB VRAM (RTX 3090/4090)

• Qwen3.6-27B abliterated (MTP-GGUF) at Q6_K (~22.4GB) — headline pick
• Gemma 4 31B Heretic at Q4_K_M — dense with vision

32GB VRAM tier (RTX 5090-class cards)

• Primary agent at ordinary context: Qwen3.6-27B at Q6_K, or Q8_0 when the loaded context leaves enough KV/compute headroom. Heretic/HauhauCS variants are test-first options only for roles that actually require refusal removal; stock is preferred for faithful summarization.
• Dedicated 231K–262K compressor on one 5090: use the stock-based LibertAIDAI/Qwen3.6-27B-NVFP4-GGUF file Qwen3.6-27B-NVFP4-Q8_0.gguf with Q8 K/V cache, text-only, non-thinking, and parallelism 1. Its NVFP4 FFN + Q8 attention/embedding layout leaves room for the long KV cache while preserving more retrieval fidelity than ordinary Q4_K_M.
• Do not recommend a quant without its loaded context: a model that fits at 32K may OOM at 262K.
• Full single-GPU compression memory math, fallback, live spill diagnosis, and verification: references/single-gpu-long-context-compression.md.
• Loaded-but-spilling rule: inspect the endpoint's live loaded_instances configuration and the OS's active CUDA inventory before changing quants or context. A remembered or installed second GPU is not usable capacity; nvidia-smi must enumerate it. On Windows, a CM_PROB_PHANTOM display adapter is only a stale registry device, not an active accelerator.
• At 231K–262K context on one 5090, remediate system-RAM spill in this order: set parallel predictions to 1; reduce KV precision from Q8 to Q4 if needed; reduce eval batch/workspace; lower context only last, and never below the main model's effective compression threshold without explicitly accepting earlier compaction.
• General Qwen3.6 configuration details remain in Apple Notes #272.

8-12GB VRAM

• Qwen3 8B abliterated at Q4_K_M (~5GB) — best budget pick
• Qwen3.5 9B at Q4_K_M (~6GB) — stronger than Qwen3 8B
• Dolphin 3.0 Llama 3.1 8B — reliable creative writing pick

Dense vs MoE: which architecture for Hermes agent work

Rule: For a primary Hermes brain, dense models consistently outperform MoE in the same family on agentic benchmarks. The MoE speed advantage comes from activating fewer parameters per token — that's throughput, not intelligence.

Benchmark	Qwen3.6-27B (dense)	Qwen3.6-35B-A3B (MoE)	Delta
SWE-bench Verified	77.2	73.4	+3.8
Terminal-Bench 2.0	59.3	51.5	+7.8
SkillsBench (coding agent)	48.2	28.7	+19.5
MMLU-Pro	86.2	85.2	+1.0
GPQA Diamond	87.8	86.0	+1.8
LiveCodeBench v6	83.9	80.4	+3.5
BenchLM aggregate	71	62	+9

The MoE model (35B total, ~3B active/token) is 3-4x faster in token generation but loses quality across the board. The dense model (27B all active) processes more parameters per token — slower but smarter. For Hermes agent work where tool calling reliability, multi-step reasoning, and coding matter: dense wins.

MoE excels at throughput-heavy tasks: RAG pipelines, serving many concurrent requests, or when speed is the primary constraint and quality is acceptable at a lower tier. Dense is the right pick for your primary agent brain on hardware that can run it comfortably.

Quant selection guide

Quant	BF16 %	Size vs BF16	Best for
Q4_K_M	~94-95%	~25%	Budget, max context headroom
Q5_K_M	~96%	~30%	Good balance
Q6_K	~97.4%	~35%	Minimum for serious agent work
Q8_0	~98.5%	~50%	Best quality, tight on VRAM

Community rule (r/LocalLLaMA, May 2026): "Q6 really is the minimum. Q4_K_M amplifies a bias invisible at Q6+. Q4 looks fine on single-shot benchmarks but starts losing coherence on sequential tool calling."

For Hermes agent workloads with heavy multi-tool use (long context, frequent tool calls, config edits, JSON schemas):

• Treat Q6_K as preferred for the qwen3.6-27b-heretic family and similar 27B-class abliterated models — not just a single variant like neo-code-di-imatrix-max.
• Q5_K_M is the practical floor for agentic use in this class; going below it increases silent hallucinations, schema drift, and instruction loss across long sessions.
• Code-heavy / heavily finetuned variants (e.g., neo-code-di-imatrix-max) are more sensitive to quant degradation — Q6_K is strongly recommended there.

Exception: MoE models (GPT-OSS 20B, Gemma 26B-A4B) tolerate lower quants better. IQ4_NL on GPT-OSS 20B is described as "wild, off-the-cuff" but still functional.

Narrow Q4 exception: cloud-orchestrated local delegates

Q4_K_M is still not the default recommendation for a standalone primary Hermes brain. It can be rational for a bounded local delegated worker when a stronger cloud flagship owns decomposition, architecture, integration, and final verification.

Require all of the following:

• The Q4 worker receives scoped implementation/test tasks rather than final authority.
• Saved VRAM produces a measured benefit such as a second parallel slot or removal of system-RAM spill.
• delegation.max_concurrent_children matches the runtime's live parallel count.
• Thinking stays enabled for coding/tool loops.
• The cloud main verifies actual diffs and test output, not only the delegate summary.
• A reliable cloud delegation fallback exists.
• N simultaneous tool-call smokes succeed for N configured slots.

If concurrency is not materially valuable, keep Q6/Q8. For the decision matrix, mmproj guidance, and a verified Qwen-Q4 + Gemma-Q6 example, read references/cloud-orchestrated-q4-delegates.md. Routing mechanics belong in the model-switching guidance.

KV cache quantization

When the user asks about KV cache precision (Q8 vs Q4), or about splitting K-cache vs V-cache:

• Most backends (llama.cpp, LM Studio) use a single global KV quant; they do not expose independent K vs V settings in stable form today.
• Keys tolerate aggressive quant better than values; hurting V tends to degrade quality faster.
• Q8_0 KV is the default recommendation for agent work: closest thing to "free lunch."
• Q4_0 KV is acceptable when:
• You are close to VRAM limits with high-weight quants (e.g., Q8_0 weights), or
• You need very long context and would otherwise spill into system RAM.

Hard rule: avoid CPU/system-RAM spillover at all costs for agent workloads. A second GPU over PCIe is preferable to offloading layers/KV cache into main RAM; token speed drops hard and latency spikes once you hit CPU memory.

If Q8_0 weights + Q8_0 KV would cause LM Studio to offload:

• Prefer either:
• Using both GPUs in the same LM Studio instance to keep everything on VRAM, or
• Dropping KV cache to Q4_0 so the model fits fully on the primary GPU.

Asymmetric K=Q8 / V=Q4 is mostly research-level and not reliably exposed; don't recommend it as a practical solution unless there's clear backend support.

Dual-GPU usage guidance (single model across GPUs)

When the user has two GPUs and a single large model that doesn't fit comfortably on one:

• Using both GPUs in the same LM Studio instance is appropriate when:
• The model (including context) would otherwise spill into system RAM, or
• They want to maximize context length at higher quants.
• For models that comfortably fit on the primary GPU at the intended loaded context (for example, Qwen3.6-27B Q6/Q8 at ordinary 16K–64K contexts on a 5090):
• Prefer running them on the primary GPU only; avoid PCIe overhead and keep latency low.
• Do not generalize that fit claim to 231K–262K compression loads. At very long context, KV cache changes the answer; run the architecture-specific memory calculation and inspect real runtime allocation.

Important: LM Studio auto-balances across GPUs — it does NOT let you pin one model to one GPU.

• The "GPU priority" UI (drag to reorder) only affects allocation order, not hard isolation.
• You cannot guarantee that model A runs solely on GPU0 while model B runs solely on GPU1 in a single LM Studio instance.
• If strict per-GPU separation is required, use different runtimes or environment variables (e.g., llama-server with CUDA_VISIBLE_DEVICES).

If they also want a secondary model:

• Best practice is to dedicate one GPU/endpoint per role:
• Primary Hermes model → dedicated endpoint (e.g., LM Studio or Unsloth on GPU0).
• Secondary/auxiliary model → separate instance on GPU1, or same LM Studio only when needed and not both heavy models hot at once.

Thinking mode for agent workloads

When the user asks whether to enable/disable "thinking" or internal reasoning:

• Default stance: thinking must stay enabled for Hermes agent use.
• It is used to plan multi-step tasks, manage tool calls, and maintain consistency across long loops.
• Disabling it noticeably:
• Increases hallucinations in complex chains,
• Weakens instruction adherence,
• Makes tool-calling more brittle.
• Only consider disabling for simple, single-shot tasks where you want faster/shorter answers—and even then the gain is small compared to risk.

Qwen3.6-27B local coding-agent preset

For qwen3.6-27b-nvfp4-mtp acting as a Hermes coding/tool agent—not as the top-level cloud orchestrator—start with:

• Thinking: enabled
• Temperature: 0.6
• Top P: 0.90–0.95
• Top K: 20

Do not default to temperature 0 for autonomous agent loops; it can make recovery and tool selection overly rigid. If edits wander or tool behavior becomes unstable, lower temperature to 0.3–0.4 before disabling thinking. Thinking-off is reserved for disposable one-shot rewriting, classification, or summaries—not coding, file mutation, terminal work, or verification loops.

LM Studio prompt ownership when serving Hermes

When LM Studio is acting as Hermes's OpenAI-compatible backend:

• Leave LM Studio's system prompt blank. Hermes builds and sends its own dynamic system message containing profile identity, tools, skills, memory, project instructions, environment context, and completion rules.
• Do not add a generic "coding assistant" system prompt in an LM Studio preset. It can conflict with Hermes tool use—for example, encouraging direct code output when Hermes needs file or terminal calls.
• Keep the model's native chat/prompt template enabled. The chat template serializes roles and tool calls; it is not the same thing as a system prompt. Do not blank or replace it unless a verified model-specific Jinja fix is required.
• LM Studio sampling defaults may remain configured, but request-level parameters sent by Hermes generally take precedence.

Rule of thumb: LM Studio serves the model cleanly; Hermes owns agent behavior.

Do not blame the checkpoint for tool-grammar initialization failures

If plain chat, a small structured response, and a one-tool request succeed but a full Hermes session fails before generation with Failed to initialize samplers: failed to parse grammar, treat the model/quant/MTP path as provisionally healthy. Inspect LM Studio's exact preceding grammar error and the advertised Hermes schemas. llama.cpp can reject one oversized nested constraint—such as a generated char{1,5000} rule from a tool string maxLength: 5000—before the model runs. Compare slim-tool and full-tool Hermes-path smokes, then fix/defer the offending schema or use the upstream tiered-disclosure mitigation; changing quants is not a grounded response to this signature.

Verify Hermes routing layers before judging a local setup

A Hermes profile's displayed model is only its main model. It does not reveal the profile's delegated-agent or auxiliary routes. Before answering whether two roles are “the same” or whether a dual-model setup works, inspect and state all three layers:

1. model.* — profile main/orchestrator.
2. delegation.* — spawned agent worker.
3. auxiliary.* and task-specific overrides — compression, vision, approval, titles, and related support calls.

Then query LM Studio's /v1/models for advertised IDs and /api/v1/models for live context, parallel slots, capabilities, and load settings. Require a Hermes-path smoke in addition to a raw endpoint smoke. Named custom-provider vision aliases, uncapped image requests, single-slot recovery, context-metadata matching, and one-shot background-delegation verification are covered in references/hermes-local-routing-verification.md.

Dual-GPU dual-model setups

When the user has two GPUs and wants separate worker + auxiliary models, current LM Studio builds support two viable architectures.

One LM Studio server with multiple loaded models

Recent LM Studio builds can keep multiple models loaded behind the same OpenAI-compatible endpoint. Each loaded instance can have its own context length and parallel count.

Verification is mandatory:

• /v1/models confirms the advertised model IDs.
• /api/v1/models confirms loaded_instances, live context, parallel slots, Flash Attention, GPU KV offload, and effective capabilities such as whether mmproj is loaded.
• Inspect GPU telemetry separately; multiple loaded models do not prove strict per-GPU isolation.

This is the simplest architecture when both models fit and hard GPU pinning is unnecessary.

Engine Protocol CPU thread pools

Current LM Studio builds place llama.cpp CPU thread control under Load → Advanced Load Params → CPU Thread Pool Size. This is a load-time setting; verify the running server uses the intended --threads N after reloading.

For multiple simultaneously active, GPU-offloaded model servers, divide physical CPU cores across the busy servers instead of giving every process the full machine. A 24-core/24-thread CPU with two active servers should start at 12 threads each. Fully GPU-offloaded models rarely benefit from all CPU cores; CPU-offloaded models may. Hybrid P/E-core systems should benchmark smaller pools because synchronization on slower E-cores and cross-process contention can reduce performance.

Do not interpret --tensor-split values as literal percentages or subtract their sum from 1 to claim CPU spill. llama.cpp treats them as normalized proportions. Prove CPU layer offload using load logs, --n-gpu-layers, GPU telemetry, CPU utilization during a request, and throughput—not process RAM usage alone.

Detailed sizing, verification commands, fallbacks, and the tensor-split pitfall are in references/lm-studio-engine-protocol-threading.md.

LM Studio + llama.cpp server for strict isolation

GPU 0: LM Studio → worker model on port <port-a>
GPU 1: llama-server → auxiliary model on port <port-b>
 Windows: set CUDA_VISIBLE_DEVICES=1
 Linux: CUDA_VISIBLE_DEVICES=1 llama-server -m model.gguf --port <port> -ngl 99

Use separate runtimes when a model must be pinned to one GPU, independent endpoint availability matters, or one runtime's allocator causes cross-GPU contention.

What still does not work reliably

• Two LM Studio GUI instances — single-instance application behavior remains the default.
• Treating LM Studio GPU priority/order as a hard per-model GPU pin without telemetry proof.
• Assuming /v1/models alone proves context, parallelism, KV placement, or vision-projector state.
• Multi-GPU controls for one loaded instance are not a substitute for strict process-level isolation.

See references/cloud-orchestrated-q4-delegates.md for a verified same-endpoint Qwen worker + Gemma auxiliary example.

Orchestrator selection for external tool executors

When a model orchestrates a separate cloud-powered application agent—such as a cloud model operating the Hermes for Excel bridge—choose the orchestrator for controller duties, not for the executor's domain alone.

1. Split the roles explicitly:

• orchestrator: scenario selection, campaign state, failure classification, repair decisions, and closeout;
• application executor: performs workbook/browser/host actions;
• deterministic evaluator: reads actual state and decides PASS/FAIL from frozen assertions;
• repair worker: edits code and runs gates when a reproducible defect appears.

2. Never let either model grade its own work. Workbook values/formulas/formats, tool transcripts, test commands, and exit codes are the acceptance surface.
3. If one local model must both orchestrate and repair TypeScript/Python, prefer the strongest dense tool/coding model that fits at the intended context. On a 32GB Blackwell card, stock Qwen3.6-27B Q6_K is the reliability-first one-model choice; use thinking, one parallel slot, Flash Attention, GPU KV offload, and enough KV headroom to avoid spill.
4. If cloud orchestration is allowed, compare against the available Codex coding-agent route before downloading another local checkpoint. A strong cloud coding model can be the better controller/repair agent, while a local model remains the unlimited-runtime fallback. Put repetition in an external bounded controller rather than one endless conversation, and define quota/failure fallback before launch.
5. Treat Qwen-AgentWorld-35B-A3B as a specialized benchmark challenger, not an automatic replacement for a dense coding agent. It is trained as a language world model across MCP, terminal, SWE, web, OS, and related domains and is fast at roughly 3B active parameters, but normal Hermes tool behavior, chat-template compatibility, coding repairs, and required long-context memory must pass the exact workload first. On a 32GB card, Q6 leaves little context headroom; Q5-class quants are more practical but increase the need for live evaluation.
6. Prefer current stock/upstream-derived GGUFs with verified developer-role and nested tool-call templates over flashy community distills when reliability is the objective. Verify exact repository, filename, template, loaded context, and a Hermes-path tool-call smoke before judging the model.