smolBSD — Complete Platform Reference

This skill provides exhaustive knowledge of the entire smolBSD framework — enough for an agent to understand, navigate, extend, and debug every aspect of the project.

Read §20 (Debugging Playbook) before touching anything related to image minimization, sailor, or login/PAM in a stripped image — these are the areas that have produced the most subtle real-world failures and are easy to misdiagnose from source alone.

1. Project Overview

smolBSD builds minimal, fast-booting NetBSD virtual machines (microVMs). Key properties:

• ~10 ms boot via PVH (PVHv2) on QEMU microvm machine type
• No prior NetBSD installation required on the host
• Immutable by design — images are built once, booted many times
• Host platforms: GNU/Linux, NetBSD, macOS (x86 VT-capable or ARM64 CPU recommended)
• Guest architectures: amd64, i386, evbarm-aarch64
• VMMs: QEMU (primary), Firecracker, Bhyve (BIOS mode)
• Images are raw .img disk files with FFS (NetBSD) or ext2 (Linux build hosts)

The fundamental unit is a service — a directory containing:

• NetBSD set selection (base, etc, comp, man, rescue, …)
• Build-time scripts (postinst/*.sh)
• Runtime init script (etc/rc)
• Build configuration (options.mk)

---

2. Project Directory Layout

smolBSD/
├── Makefile              # Entry point for manual image building (bmake)
├── mkimg.sh              # Image creation script (called by Makefile, not directly)
├── startnb.sh            # Low-level QEMU VM launcher
├── smoler.sh             # High-level CLI dispatcher: build|run|push|pull|images
├── batch.sh              # Batch launcher: N copies of a service on shifted ports
├── smoler/
│   ├── build.sh          # SMOLerfile parser → generates service dir + calls bmake
│   └── img.sh            # OCI push/pull/list (oras wrapper)
├── scripts/
│   ├── app-run.sh        # Launcher helper for the app/ GUI
│   ├── fetch.sh          # Smart curl wrapper (globbing, fresh checks)
│   ├── freshchk.sh       # Freshness check (remote Last-Modified cache in db/)
│   ├── sh                # Static shell used as /rescue/sh in rescue images
│   └── uname.sh          # Architecture/machine detection helper
├── sailor/               # Cloned sailor repo (minimization), invoked by mkimg.sh
├── service/              # All service definitions
│   ├── common/           # Shared runtime scripts bundled into /etc/include/ in VM
│   │   ├── basicrc       # Standard env, networking, devices, SSL_CERT_FILE, rc.pre/rc.local
│   │   ├── choupi        # Emoji/ASCII toggle for terminal output
│   │   ├── funcs         # rsynclite() helper (tar-based directory sync)
│   │   ├── vars          # BASEPATH, DRIVE2 path constants
│   │   ├── shutdown      # Clean halt (sync, umount, optional viocon kill signal)
│   │   ├── mount9p       # 9P filesystem mount (host directory sharing)
│   │   ├── qemufwcfg     # QEMU fw_cfg variable loader
│   │   ├── pkgin         # Package manager bootstrapper
│   │   └── sailor.vars   # Sailor integration variables
│   ├── base/             # Full base+etc system with ksh (builder image base)
│   ├── build/            # Builder microVM service (orchestrates service builds)
│   ├── rescue/           # ~10 MB minimal rescue shell
│   └── <service>/        # One directory per service
│       ├── etc/rc        # Runtime init script (MANDATORY for init(8) services)
│       ├── postinst/     # Build-time scripts executed on host/VM builder
│       ├── options.mk    # Service build variables (IMGSIZE, ADDPKGS, SETS, etc.)
│       ├── own.mk        # User overrides (git-ignored, not committed)
│       ├── sailor.conf   # Sailor minimization rules
│       ├── packages/     # Pre-built binary packages for offline install
│       └── NETBSD_ONLY   # Marker: build only on native NetBSD
├── smolerfiles/          # SMOLerfile / Dockerfile examples
│   ├── Dockerfile.inc    # Shared INCLUDE snippets
│   ├── Dockerfile.<name> # Per-service SMOLerfile (Dockerfile-compatible)
│   ├── SMOLerfile.<name> # Named SMOLerfiles (same syntax, different naming)
│   ├── *.smol            # Minimal SMOLerfiles (service name from filename)
│   └── *.inc             # Shared include fragments
├── etc/                  # VM config files for startnb.sh (-f flag)
│   └── <service>.conf    # hostfwd, imgtag, use_pty, KERNEL, NBIMG, etc.
├── bios/                 # BIOS firmware files for microvm machine type
├── confkerndev/          # Kernel driver disabler tool (SMOLIFY)
├── app/                  # Flask-based web GUI for VM management
├── www/                  # Project website and assets
├── k8s/                  # Kubernetes device plugin / deployment examples
├── misc/                 # Miscellaneous documentation
├── contribs/             # Contributed scripts
├── share/                # Shared assets (e.g. ssh.pub keys)
├── .github/workflows/    # CI/CD pipeline
│   ├── main.yml          # Builder + rescue images for amd64 + evbarm-aarch64 on push
│   └── smoler.yml        # SMOLerfile service images on smolerfiles/* push
├── db/                   # Fetch freshness cache (remote Last-Modified, see freshchk.sh)
├── images/               # Built .img disk images (empty in repo, populated at build)
├── kernels/              # Downloaded kernels (empty in repo, populated at build)
├── sets/                 # Downloaded NetBSD sets (empty in repo, populated at build)
├── pkgs/                 # Optional pre-fetched packages (empty in repo, populated at build)
├── mnt/                  # Build-time mount point (empty directory)
└── disks/                # Additional disk images

---

3. Two Workflows

3.1 smoler.sh (Docker-style, high-level)

smoler.sh is a thin dispatcher that routes subcommands to dedicated scripts:

Command	Routes To	Purpose
./smoler.sh build [-y] [-t tag] [--build-arg K=V] [VAR=val] <SMOLerfile>	smoler/build.sh	Parse SMOLerfile → generate service dir → call bmake build
./smoler.sh run <image> [startnb.sh flags]	startnb.sh	Run a built image (resolves name → config file or raw path). Any startnb.sh flag passes through after the image name (e.g. -l drive2,drive3 for extra drives, -r, -e).
./smoler.sh push <image>	smoler/img.sh	Push to OCI registry via oras
./smoler.sh pull <image>	smoler/img.sh	Pull from OCI registry via oras
./smoler.sh images [ok]	smoler/img.sh	List local images with size, date, signature status. Columns are sized as fixed proportions of terminal width (name = 50%, size/date/sig = 25% each); not auto-fit to content. Pass ok to show only images with verified smolsig.

smoler.sh run name resolution:

1. Strips -amd64:… or -evbarm-aarch64:… suffix to get base service name (note: an -i386:… suffix is not stripped — the etc/<base>.conf lookup for i386 images is a known quirk; the images/<image>.img fallback still works with the full name)
2. Checks for etc/<base>.conf → passes -f etc/<base>.conf to startnb.sh
3. Falls back to checking images/<image>.img → passes -i <image> to startnb.sh
4. If neither exists, shows startnb.sh -h usage

smoler.sh build regeneration: if service/<name>/ already exists, build.sh deletes its etc/rc, options.mk, postinst/ and etc/<name>.conf before regenerating them. Untracked files in the service dir (e.g. sailor.conf, own.mk) survive, but anything hand-edited in the deleted files is lost — commit what matters first.

3.2 bmake / make (Manual, low-level)

Command	Purpose
bmake buildimg	Build the builder image (native on NetBSD/FreeBSD/Linux; on macOS this fails — the build target falls back to fetchimg there)
bmake fetchimg	Download pre-built builder image from GitHub Releases (macOS, no FFS support)
bmake SERVICE=<name> build	Build a service image using the builder microVM
bmake SERVICE=<name> base	Build only the base filesystem (no builder VM — runs mkimg.sh directly)
bmake SERVICE=<name> MOUNTRO=y build	Build with read-only root
bmake SERVICE=<name> ARCH=evbarm-aarch64 build	Build for ARM64
bmake kernfetch	Download the appropriate kernel
bmake setfetch	Download NetBSD sets
bmake pkgfetch	Download binary packages
bmake fetchall	All of the above
bmake rescue	Shortcut: SERVICE=rescue build
bmake live	Fetch a full NetBSD live image

Platform-specific builder image behavior (Makefile build target):

• On NetBSD/FreeBSD/Linux: builds the builder image natively (bmake buildimg)
• On macOS (and any other OS): fetches the pre-built builder image from GitHub (bmake fetchimg); running buildimg directly on macOS fails because mkimg.sh rejects macOS
• Builder image freshness is checked via SHA256; rebuilds/fetches only when the remote changes

---

4. SMOLerfile / Dockerfile Reference

SMOLerfiles are nearly 100% Dockerfile-compatible. smoler/build.sh parses them line-by-line and generates:

• service/<name>/options.mk — build variables
• service/<name>/etc/rc — runtime init script
• service/<name>/postinst/postinst-N.sh — build-time execution scripts
• etc/<name>.conf — VM config for startnb.sh

4.1 Parsing Flow (build.sh internals)

1. INCLUDE expansion: INCLUDE <file> directives are resolved first by catting the referenced file inline, producing a flat temporary SMOLerfile
2. LABEL extraction: All LABEL lines (with or without smolbsd. prefix) are extracted via sed/awk, uppercased, and written to options.mk
3. Service name: From LABEL smolbsd.service=NAME, or from .smol filename (SMOLerfile.foo → SERVICE=foo)
4. Postinst-0.sh: Generated with chroot setup (pkgin bootstrap, resolv.conf, openssl certs)
5. Line-by-line parsing: Each directive generates shell commands appended to postinst scripts or etc/rc
6. Finalization: etc/rc gets . /etc/include/shutdown appended; etc/<name>.conf gets imgtag and use_pty
7. Build: Calls make (NetBSD) or bmake (elsewhere) with SERVICE=<name> IMGTAG=:<tag> build

4.2 All Supported Directives

Directive	Syntax	Description
FROM	FROM base,etc or FROM base-amd64.img	Set names or an existing image name. If omitted, the Makefile SETS default (base,etc) is used.
LABEL smolbsd.service=NAME	LABEL smolbsd.service=caddy	Mandatory. Sets the service name.
LABEL smolbsd.imgsize=N	LABEL smolbsd.imgsize=2048	Image size in MB (default: 512).
LABEL smolbsd.minimize=y	LABEL smolbsd.minimize=y	Shrink to actual usage + 10%. MINIMIZE=+N adds N MB instead. See §13 and §20.1 before combining with WAPBL.
LABEL smolbsd.publish="H:G"	LABEL smolbsd.publish="8881:8880,2289:22"	Port mappings (host:guest), comma-separated.
LABEL smolbsd.use_pty=y	LABEL smolbsd.use_pty=y	Use PTY console (needed for interactive apps like vim/tmux).
LABEL smolbsd.addpkgs="pkg1 pkg2"	LABEL smolbsd.addpkgs="pkgin curl"	Packages to fetch/untar at build time (no pkgin needed).
RUN	RUN pkgin up && pkgin -y in caddy	Execute commands during build (chrooted). Supports heredocs (<<EOF).
ARG	ARG FOO=bar	Build argument with optional default. Override with --build-arg FOO=val.
ENV	ENV NBUSER=clawd	Set environment variable (available in build scripts and /etc/rc).
EXPOSE	EXPOSE 8880	Document exposed ports. Requires smolbsd.publish LABEL for actual mapping — or use the non-Docker shorthand EXPOSE 8881:8880 (host:guest) which maps ports directly.
USER	USER clawd	Switch user for subsequent RUN, CMD, and COPY ownership.
WORKDIR	WORKDIR /home/clawd	Set working directory. Adds cd to /etc/rc and becomes the cwd for all RUN commands.
CMD	CMD caddy respond -l :8880	Default command to run at boot (appended to /etc/rc).
ENTRYPOINT	(same syntax as CMD)	Treated identically to CMD in smolBSD.
COPY	COPY src dest	Copy files from build context into image. Supports --chown, --chmod, --exclude.
ADD	ADD url dest	Like COPY but also supports HTTP(S) URLs (fetched via ftp).
VOLUME	VOLUME /data	Declare a host directory mount point. Writes share= to config.
SHELL	SHELL ["/bin/bash", "-c"]	Change the shell used for RUN instructions. The -c flag is stripped. Creates a new postinst script.
INCLUDE	INCLUDE Dockerfile.inc	smolBSD extension. Inline the contents of another file.

4.3 FROM — Set Selection Details

FROM base,etc                    # Standard: base system + /etc config files
FROM base,etc,man,comp           # Full: adds man pages and compiler toolchain
FROM comp:/usr/bin/strip         # Partial: only extract /usr/bin/strip from comp set
FROM comp:/usr/libexec/*         # Glob: extract matching files from comp set
FROM base-amd64.img              # Inherit from a pre-built image

Valid set names: base, etc, man, comp, rescue, games, modules, tests, text, xbase, xcomp, xetc, xfont, xserver.

4.4 RUN — Heredoc Support

RUN <<EOF
hostname myhost
ulimit -n 4096
echo 'eval \$(resize)' >> /etc/rc.local
EOF

The parser detects <<EOF (or any tag) and appends lines until the closing tag. Quotes around the tag are stripped. Heredoc content is escaped (" → \") before being wrapped in chroot . su ${USER} -c "...".

4.5 COPY / ADD — Options

COPY --chown=clawd --chmod=600 /host/ssh.pub /home/clawd/.ssh/authorized_keys
ADD --exclude=.git ./src /app

• --chown=user:group or --chown=user — set ownership via chown -R in chroot
• --chmod=mode — set permissions via chmod -R in chroot
• --exclude=pattern — passed to rsynclite (tar-based sync)
• HTTP(S) URLs in ADD/COPY are fetched via ftp -o
• Destination paths starting with $ are treated as variable references

4.6 Generated etc/.conf Format

hostfwd=::8881-:8880,::2289-:22
imgtag=latest
use_pty=y
share=/host/path      # from VOLUME

4.7 Postinst Script Numbering

The parser generates numbered postinst scripts:

• postinst-0.sh — chroot bootstrap (pkgin setup, resolv.conf, openssl certs)
• postinst-1.sh — first RUN/COPY/ADD/USER/VOLUME/WORKDIR block (default shell)
• postinst-N.sh — new script created when SHELL directive changes the shell
• postinst.args — accumulated ARG/ENV exports shared across scripts

4.8 File Naming Conventions

• Dockerfile.<name> — standard Dockerfile naming; service name from LABEL smolbsd.service
• SMOLerfile.<name> — same syntax; service name from LABEL smolbsd.service
• <name>.smol — minimal files; service name extracted from filename itself
• *.inc — include fragments (used with INCLUDE directive)

4.9 make vs bmake

bmake is the host-side build tool (required on Linux/macOS; on NetBSD it's synonymous with make). It invokes the top-level Makefile targets (build, buildimg, base, …).

Inside the builder VM (i.e., in RUN directives and postinst/*.sh scripts), the environment is NetBSD — use plain make, not bmake. The builder VM includes make from the comp set; bmake is not guaranteed to be available.

# Wrong (bmake is a host tool, not inside the VM):
RUN cd /tmp/src && bmake && bmake install

# Correct (plain make inside the NetBSD builder VM):
RUN cd /tmp/src && make && make install

---

5. Service Directory Manual Reference

5.1 options.mk — All Known Variables

Variable	Type	Default	Description
SERVICE	string	(target name)	Service name, determines output filename
IMGSIZE	int	512	Image size in megabytes
SETS	string	base.${SETSEXT} etc.${SETSEXT}	NetBSD sets to include (space-separated)
ADDSETS	string	(empty)	Additional sets beyond SETS
ADDPKGS	string	(empty)	Packages to fetch and extract into image
MINIMIZE	y/+N	(empty)	y = +10%, +512 = explicit MB to add
MOUNTRO	y	(empty)	Mount root read-only (-o passed to mkimg.sh)
BIOSBOOT	y	(empty)	Enable BIOS boot (GPT + bootxx_ffsv1)
BIOSCONSOLE	string	com0	Console device for BIOS boot (com0, pc)
SMOLIFY	y	(empty)	Run confkerndev to disable unused kernel drivers
FROMIMG	string	(empty)	Inherit from existing image instead of sets
PKGVERS	string	11.0	Package version for pkgsrc URL
ARCH	string	(detected)	Target architecture: amd64, i386, evbarm-aarch64
CURLSH	string	(empty)	URL to a shell script executed as finalizer
SETSEXT	string	tar.xz	Set archive extension (tgz for i386)
IMGTAG	string	(empty)	Suffix appended to image name (e.g. :latest)
SVCIMG	string	(empty)	When set, only run postinst/<SVCIMG>.sh
PUBLISH	string	(empty)	Port mappings (used by SMOLerfile parser for EXPOSE)

Conditional variables (Makefile syntax in options.mk):

.if defined(MINIMIZE) && ${MINIMIZE} == y
ADDPKGS=pkgin pkg_tarup pkg_install sqlite3 rsync curl
.endif

5.2 etc/rc — Runtime Init Script

This is the heart of every service. Standard structure:

#!/bin/sh

. /etc/include/basicrc          # Mandatory: env, networking, devices
. /etc/include/mount9p          # Optional: host directory sharing

# tmpfs mounts for writable overlays
mount -t tmpfs -o -s10M tmpfs /tmp
mount -t tmpfs -o -s10M tmpfs /var/log
mount -t tmpfs -o -s1M tmpfs /var/run
mount -t tmpfs -o -s10M -o union tmpfs /etc

# Service-specific setup (users, permissions, config)
useradd -m sshd
mkdir -p /home/sshd/.ssh

# Start services
/etc/rc.d/sshd onestart

# Main command (blocks until service exits)
exec myapp

. /etc/include/shutdown         # Clean halt

Key hooks in basicrc:

• /etc/rc.pre — custom pre-boot hook (sourced before device setup)
• /etc/rc.local — custom post-boot hook (sourced after networking, before MOUNTRO)
• SSL_CERT_FILE env var — if set, copies custom SSL certs and runs certctl rehash

5.3 postinst/*.sh — Build-Time Scripts

These execute on the build host (or builder VM) inside the mounted image root. Use for:

• Downloading external binaries with curl or ftp
• Extracting archives
• Setting up chroot environment
• Pre-configuration that doesn't need pkgin

They are NOT run inside the microVM at boot time.

Important conventions:

• Scripts run from the mounted image root (i.e., pwd is the fake root)
• Paths like etc/ssh/ refer to the image's /etc/ssh/
• Use ../service/<name>/etc/ to access files from the service directory
• Source ../service/common/funcs for rsynclite() and ../service/common/choupi for emoji output
• Check /BUILDIMG marker file to verify running inside the builder VM

5.4 own.mk — User Overrides

Not committed to git. Same format as options.mk. Loaded after options.mk so it overrides. Use for personal dev settings.

# service/myapp/own.mk (git-ignored)
IMGSIZE=1024
ADDPKGS=pkgin curl vim

5.5 sailor.vars (in service/common/)

Seed variables consumed by sailor when mkimg.sh runs minimization (see §13.2). Current contents:

• shippath — the smolBSD build drive path (/drive2), where sailor finds the image being minimized
• shipbins — baseline list of binaries always kept (init, mount, sh, useradd, login, /usr/lib/security/*, …)
• sync_dirs — directories kept in sync rather than stripped (/etc, certs, pkgin config, terminfo, zoneinfo, /var/log)
• packages — package names treated as ship targets (curl, rsync)

Relationship: treat sailor.vars as the floor, and per-service sailor.conf as the diff on top of it. If a stripped image later fails in surprising ways (WAPBL errors, missing PAM modules, broken login), the fix is almost always to add a keep-rule in sailor.vars or the service's own sailor.conf, not to patch mkimg.sh — see §20.

5.6 packages/ — Offline Binary Packages

Place pre-built .tgz packages here. mkimg.sh rsyncs them to the image root as /packages/. The pkgin common script detects /packages/ and installs them via pkg_add.

5.7 NETBSD_ONLY — Platform Marker

If this empty file exists, mkimg.sh refuses to build on non-NetBSD hosts:

This image must be built on NetBSD!
Use the image builder instead: make SERVICE=<name> build

---

6. Build Pipeline — Deep Dive

6.1 Image Creation (bmake SERVICE=foo build)

The build target in the Makefile orchestrates a two-stage process:

Stage 1: Builder microVM creation

bmake buildimg

1. SERVICE=build IMGTAG= base — calls mkimg.sh to create images/build-amd64.img
2. Extracts base + etc sets (plus partial comp:/usr/bin/strip) with MOUNTRO=y
3. Creates FFS (NetBSD) or ext2 (Linux) filesystem on the image
4. Installs the builder's own /etc/rc that waits for a second drive and executes build commands

Stage 2: Service build inside builder VM

bmake SERVICE=foo build

1. fetchall — download sets, packages, and kernel
2. Creates a blank disk image of IMGSIZE MB (via dd)
3. Writes ENVVARS to tmp/build-foo (lock/coordination file)
4. Launches the builder VM with startnb.sh:
   • -k kernels/netbsd-SMOL — PVH kernel
   • -i images/build-amd64.img — builder rootfs
   • -l images/foo-amd64.img — second drive (target image; -l also accepts a comma-separated list for multiple extra drives)
   • -w . — 9P share of project directory
   • -p ::22022-:22 — SSH access
   • -c $BUILDCPUS -m $BUILDMEM (defaults 2 cores / 1024 MB)
   • -x "-pidfile qemu-<service>.pid"
5. Builder VM's /etc/rc detects the second drive, sources tmp/build-foo, calls make base to invoke mkimg.sh to populate the target image
6. Builder removes tmp/build-foo when done (a final cat keeps the VM alive after)
7. Host polls the lock file, then kills builder QEMU via the pidfile
8. If MINIMIZE is set, waits for the image to be released (lsof), then resizes via qemu-img resize --shrink $(cat tmp/<img>.size) — if the image also uses WAPBL journaling, see §20.1
9. Writes signature to image and .sig file: smolsig:DD/MM/YYYY|UUID

6.2 mkimg.sh — Internal Flow