GPU Fleet & Role Assignment

Overview

Sound Suite splits its AI work into modes — discrete jobs like embedding, OCR, reranking and deep reasoning. Each runs independently, needs a different amount of memory, and can live on a different machine.

That design exists to answer one question well:

Keep it local where you can. Where you can't, route it through OpenRouter — except OCR, which is always local.

You are not choosing between "everything on my hardware" and "everything in the cloud." You choose per mode, and you can change your mind later without reindexing.

Four admin screens control this:

Screen	Answers
GPU Fleet	Which machines are connected, and are they healthy?
Mode Types	What jobs exist, what model does each run, and where can it run?
Role Assignments	Which runtime serves each mode on each host?
Host Provisioning	Per-host overrides — OS, master URL, ports

The modes

Seven modes, each a separate job with its own model and memory footprint:

Mode	What it does	Default model	VRAM
`ss-embedding`	Document and query embedding. Used in both indexing and search	`qwen3-embedding:4b-fp16`	~1.2 GB
`ss-code-embedding`	Code-aware embedding for agent and code search, kept separate from text	`qwen3-embedding:4b-fp16`	~1.2 GB
`ss-completion`	Chat completion at search time	`qwen3.5:9b`	~10 GB
`ss-ocr`	Visual OCR for low-density pages and exhibit images, during indexing	`AuditAid/PaddleOCR-VL-1.6-0.9B`	~5 GB
`ss-reranker`	vLLM cross-encoder reranking	`Qwen/Qwen3-Reranker-8B`	—
`ss-rlm`	Recursive Language Model for deep long-context reasoning across many documents	`mit-oasys/rlm-qwen3-8b-v0.1`	~10 GB
`ss-rlm-sandbox`	Fallback for `ss-rlm` — the same recursive pattern in a sandboxed Python REPL, driven by a hosted model	via OpenRouter	CPU only

Two things worth noticing.

Embedding is cheap; completion is not. At ~1.2 GB, embedding fits almost anywhere. Completion and RLM want ~10 GB each. Splitting them across machines is the entire point of a fleet — you don't need one card that can hold everything.

ss-rlm-sandbox is the escape hatch. ss-rlm runs a Qwen3-8B fine-tune that no provider hosts, so it cannot be moved to the cloud as a model. But RLM is an inference strategy, not a weight — so the sandbox runs the same recursive pattern in a CPU-only container and calls a hosted chat model for each step. Deep Search on a machine with no GPU at all.

Routing: the local-to-cloud spectrum

Each mode gets one of four routing policies:

Policy	Behaviour
SideCar Only	Local hardware or nothing. Never calls out
SideCar + OpenRouter Backup	Prefer local; fall back when no host is serving it
SideCar + OpenRouter	Use both as available
OpenRouter Only	Skip local entirely

This is the setting that makes the whole system flexible. A firm with one GPU server might run embedding and OCR locally and route completion to OpenRouter. A solo practitioner on a laptop might run OpenRouter Only for everything except OCR.

Defaults are conservative. Most modes default to SideCar Only — nothing is routed outward until you choose it. Reranking is the exception: it defaults to Backup, because the previous behaviour was worse than a fallback. vLLM needs CUDA, so every Mac in a fleet simply went without reranking, and search results silently came back in a worse order.

OCR has no routing policy at all. It is absent from this mechanism, not merely set to off. Why.

Capping what routed work can cost

Every role that can route outward also gets a daily spend cap in USD, set per role on the same screen. Blank or 0 means uncapped.

Per-role matters here: a large ingest that runs embedding hard cannot drain the budget your chat model needs. When a role hits its cap it reverts to local for the rest of the day and says so, rather than failing silently, and counters reset at UTC midnight so the boundary is identical across a fleet.

A circuit breaker covers the case a cap cannot — a retry loop against a provider that is erroring still bills. Consecutive failures open the circuit for a cooldown.

Spend controls in detail →

GPU Fleet

The fleet screen lists every connected sidecar with hostname, URL, connection type, status and last-seen time.

Per-model idle timeouts. Each mode — Embedding, Code Embedding, Completion, OCR, Reranker, RLM — gets its own idle timeout. A model sitting unused past its timeout is unloaded and its VRAM released. Tune them by access pattern: embedding is touched on every search and may as well stay resident; OCR runs in indexing bursts and can be evicted aggressively.

Minimum online instances. Keep N hosts serving a given mode at all times, so a mode you depend on doesn't vanish because one machine slept.

Runtime demand is reported over a peak five-minute window — what was actually asked for, rather than what you assumed. Useful before deciding a machine is unnecessary.

Save & Push to All Sidecars distributes configuration fleet-wide, so you set idle timeouts once rather than per host. There is also Sync Models, and a Communication Log for diagnosing a host that registers but never receives work.

Mode Types

A read-only catalogue: every mode, its current default model, and the platforms it can run on.

Its real job is showing you where a mode cannot go. Availability is model-aware, not just OS-aware — so changing the OCR model can make a Mac host ineligible for ss-ocr, and the catalogue reflects that immediately rather than letting you assign work that will never be served.

The reranker is the clearest case: it is vLLM, which needs NVIDIA, so it is Linux-native or Windows Docker with GPU passthrough. On a Mac the honest answer is the OpenRouter fallback.

Role Assignments

A grid of modes against runtimes. For each mode on each host, pick which runtime serves it:

Runtime	Where it fits
Ollama (native)	Apple Silicon — Metal, no container overhead
Docker Ollama	Linux and Windows WSL2
Docker vLLM	NVIDIA passthrough; what the reranker needs
Docker Model Runner	Docker Desktop's runner — on Mac this means vllm-metal
Docker (no GPU)	CPU containers — how `ss-rlm-sandbox` runs
Off	Not served here

Same mode, different runtime per machine. ss-embedding might run on native Ollama on a Mac and Docker Ollama on a Linux box, and the master routes to whichever is healthy.

Sync now pushes assignments to the fleet; Reset to defaults restores the OS-appropriate baseline.

Host Provisioning

Per-host overrides for cases where detection isn't right:

OS override — Auto, Mac Docker (Ollama), Windows Docker (WSL2), or Linux. Auto detects; the override is for when it guesses wrong or you want a host treated differently
Master URL (per-host) — which master a sidecar reports to, for multi-master setups
WS port — the websocket port for master↔sidecar traffic
Sidecar URL — read-only, as registered
Notes — free text, so "the GPU box in the server room" is recorded somewhere

Putting it together

Three shapes this supports:

No GPU. Everything except OCR routed OpenRouter Only; OCR on the CPU engine. Deep Search works via ss-rlm-sandbox. No hardware purchase.

One GPU machine. Embedding (~1.2 GB) and OCR (~5 GB) local on that box, with completion and RLM set to SideCar + OpenRouter Backup — local when the machine is free, hosted when it's busy or asleep.

A real fleet. Modes spread across machines by memory footprint. Everything SideCar Only, nothing leaving the network, with per-model idle timeouts freeing VRAM between jobs.

All three run the same software. The difference is a handful of dropdowns.