Edge / On-Device Inference

Why on-device (the decision drivers)

• Latency: on-device token/inference in ~tens of ms vs 200–500ms cloud round-trip — decisive for voice, AR, control loops.
• Privacy / offline / cost: data never leaves the device; works without connectivity; no per-call cloud cost.
• Cost of doing it: tight memory/energy/thermal budgets; smaller models; more engineering.

The hardware tiers (match model to silicon)

Tier	Silicon	Typical model
MCU / TinyML	Cortex-M + tiny NPU (sub-$1 class)	KB-MB models: keyword spotting, anomaly detection (TFLite Micro)
Mobile / AI-PC	Phone NPU, laptop NPU (tens of TOPS)	Quantized 3–8B LLMs, vision
Edge box	Jetson Orin/Thor, Coral, Hailo (40+ TOPS)	7–13B LLMs, multi-camera vision

Architectural levers

• Quantization is the key enabler: FP16 → INT8 → INT4 trades accuracy for memory/throughput/energy. Most edge LLMs run INT4/INT8. Validate accuracy loss.
• Model choice: small instruct/distilled models (e.g. 3–8B class) and vision models sized to the NPU's memory bandwidth, not just TOPS.
• Runtime: TFLite/LiteRT, ONNX Runtime, ExecuTorch, llama.cpp/Ollama, vendor SDKs (TensorRT for Jetson). Pick what targets the accelerator.
• Budgets: state the TOPS, RAM, and energy-per-inference budget up front — they bound everything. Batch=1, KV-cache memory dominates LLM RAM.

When to recommend edge vs cloud vs hybrid

• Hard latency / offline / privacy / per-unit cost → on-device.
• Large model / variable load / centralized updates → cloud serving.
• Both needs → hybrid (small local model + cloud escalation): see hybrid-edge-cloud.