OpenRouter Integration

Overview

Sound Suite was built to run on your own hardware. That remains the default and the point. But "runs locally" historically meant "needs a GPU" for several roles — and a 10 GB VRAM requirement is a hard barrier if you have a laptop and a case to work.

The OpenRouter integration removes that barrier for the roles where it can be removed safely, without moving your documents anywhere.

Every inference role can now run through OpenRouter — except OCR.

Role	Routes to OpenRouter?	Default mode
Embedding	✅	`local-only`
Code embedding	✅	`local-only`
Completion	✅	`local-only`
Reranker	✅	`local-first`
Deep Search (RLM)	✅	`local-only`
OCR	❌ never	local, always

OCR isn't merely switched off — it has no routing mode at all. The mechanism that sends work to OpenRouter does not exist for it, by design. Reasoning below.

The headline consequence: Deep Search no longer needs a GPU. What used to require a self-hosted model at roughly 10 GB of VRAM now runs in a CPU-only container that drives the same recursive-reasoning pattern against a hosted chat model.

Every role defaults to local. Nothing is routed outward until you configure it.

The RLM sandbox: Deep Search without a GPU

What RLM is

Deep Search is powered by a Recursive Language Model. Rather than stuffing a whole corpus into one enormous prompt, the context is held as a variable in a REPL that the model programmatically chunks, greps and reasons over — calling itself recursively on the pieces that matter.

The key insight, and the reason this integration is possible at all:

RLM is an inference strategy, not a weight.

The pattern replaces llm.completion(prompt, model) with rlm.completion(prompt, model). Any competent chat model can drive it.

Why that mattered

The self-hosted role, ss-rlm, runs a Qwen3-8B fine-tune post-trained for recursive reasoning, on Docker vLLM at roughly 10 GB of VRAM. No provider hosts that specific model — it is absent from OpenRouter's catalogue entirely. It is one of only two roles on the fleet that no hosted service can serve directly.

So the choice looked like: own a suitable GPU, or go without Deep Search.

What shipped

ss-rlm-sandbox is a container built FROM python:3.11-slim. No CUDA, no GPU, no model weights. It runs the RLM pattern and calls out to a hosted chat model for each sub-completion.

┌──────────────────────┐
│  ss-rlm-sandbox      │   CPU-only · python:3.11-slim
│  RLM pattern in a    │   no GPU, no weights, no API key
│  sandboxed REPL      │
└──────────┬───────────┘
           │ sub-model calls
           ▼
┌──────────────────────┐
│  Sidecar             │   holds the OpenRouter key
└──────────┬───────────┘
           │
           ▼
      OpenRouter → hosted chat model

Deep Search now runs on a machine with no discrete GPU at all.

The sandbox is locked down on purpose

The Python executing inside that container is written by a model, not by us. That makes three properties requirements rather than preferences:

Property	Why
No Docker socket	`/var/run/docker.sock` is never mounted in. The REPL runs with a local environment rather than the library's Docker-in-Docker mode, precisely so the socket is never needed
No API key inside	The container holds no credentials. Sub-model calls route back out through the sidecar, which is what holds the OpenRouter key
No outbound egress	The only route the container needs is to the sidecar. Everything else is closed

The RLM library itself is vendored at public/rlm/ and served from disk by any master. A build host needs a route to a master and nothing else — not GitHub, not PyPI — and the tarball's SHA-256 is verified on the way in, the same way the sidecar's own installer verifies its release.

Choosing where RLM runs

The routing mode is a configuration setting, and the default is conservative:

Mode	Behaviour
`local-only`	Default. Use the self-hosted `ss-rlm` or nothing. No sidecar running it means Deep Search is unavailable — never a silent cloud call
`local-first`	Try `ss-rlm`; fall back to the sandbox if no host is serving it
`cloud-only`	Go straight to the sandbox; never probe for a local `ss-rlm`

Nothing routes to OpenRouter until you turn it on. A master with no pushed OpenRouter configuration simply never uses it.

Embeddings through OpenRouter

Embedding generation can be pointed at OpenRouter instead of a local model. Two details in the implementation are worth understanding, because both protect your index from a failure mode that produces no error.

Provider pinning

OpenRouter may route a single model id to any of several upstream providers. Two providers serving the same model do not guarantee identical vectors.

An unpinned call could therefore reroute partway through a corpus and quietly split one logical vector space in two — degrading recall with nothing in any log to explain it. Every embedding call pins its provider explicitly, taken from a curated catalogue entry.

Dimension safety, failing closed

The expected vector width is passed through to the client, which throws before returning if the response width does not match.

This matters more than it sounds. The vector store reacts to a schema mismatch by dropping and recreating the table — so a wrong-width vector reaching a write path could destroy an existing corpus. The check fails closed, before any write occurs.

Supported models

Only models from the curated embedding catalogue are accepted — currently the qwen3-embedding family plus the OpenAI and Google embedding models OpenRouter proxies. Anything else throws at construction rather than at write time.

Dimensions come from the catalogue entry, which records measured widths rather than documented ones.

Reranking

Reranking has three settings: vllm, openrouter, or none.

Unlike the other roles, reranking defaults to local-first rather than local-only — it uses a local vLLM host when one is configured, and falls back to a hosted reranker otherwise. That default exists because the previous behaviour was worse than a fallback: every Mac in a fleet simply went without reranking, since vLLM needs CUDA.

Reranking is the stage most sensitive to being silently absent. Results still come back, just in a worse order, and nothing tells you. Search results carry an explicit signal when they were not reranked.

Completions

Chat, draft assistance and summarisation can route through OpenRouter the same way, with local-first and hybrid modes available in addition to local-only and cloud-only.

OCR stays local. Permanently.

OCR runs as the ss-ocr fleet role — AuditAid/PaddleOCR-VL-1.6-0.9B, roughly 5 GB of VRAM — handling low-density PDF pages and exhibit images during indexing. It can also fall back to a CPU engine on the master. Either way it is your hardware.

There is no cloud option for it, and there will not be one. This is a design decision, not a gap:

The model has no hosted equivalent. PaddleOCR-VL is not in OpenRouter's catalogue. No hosted model is the same model.
A generic vision model returns a different output shape. The quality gate that rejects bad OCR — repetition loops, letter soup, LaTeX recitation — is tuned against PaddleOCR's structure. Feed it a general-purpose VLM's prose and the gate's verdicts become meaningless. You would lose the ability to detect bad OCR, which is worse than slower OCR.

The practical consequence is the one that matters: your page images are never transmitted anywhere. The most sensitive artefact in the pipeline — a rendered image of an actual filed document — stays inside your network under every configuration.

Administration

The OpenRouter tab provides:

Model catalogue — browse the full hosted catalogue, filtered to what each role can actually use
Credits — current balance
Activity — recent spend, so cost is observable rather than inferred at month end
Validate — check a key before relying on it
Settings — per-role model selection, including the sandbox model Deep Search drives

Spend controls

Routing work to a metered API changes the failure mode. A bug that previously wasted CPU now bills you. Three controls exist because of that, and they were required before any non-local mode was allowed in production.

Why they exist

During development, a background loop was measured acquiring a resource every 30 seconds, forever, with zero releases.

Against a local counter, that inflates a number and someone notices eventually. Against a metered API, the identical bug bills — quietly, continuously, at whatever rate the loop runs. The spend guard exists because that bug was observed, not because it was imagined.

Per-role daily caps

Each role gets its own daily cap in USD. 0 or blank means uncapped.

Caps are per role, not global, so runaway embedding during a large ingest cannot consume the budget your chat model needs. Set a cap where you want a ceiling and leave the rest open.

When a role reaches its cap, the call is refused with an explicit message naming the role, the amount spent and the limit:

OpenRouter daily cap reached for role "embedding":
$4.9812 of $5.00 — reverting to local

The role reverts to local for the rest of the day rather than failing silently, and the event is logged loudly. Counters reset at UTC midnight — a fixed boundary rather than a rolling window, so "today's spend" means the same thing on every host in a fleet regardless of timezone.

The guard is enforced in the shared client that every caller goes through, which is what makes one cap cover embeddings, reranking and chat alike. No provider implements its own.

Circuit breaker

A cap limits successful spend. It does nothing about a retry loop against a failing provider — which bills for errors.

So N consecutive upstream failures opens a circuit for a cooldown period, and calls are refused while it's open. Without this, a provider having a bad hour plus an automatic retry becomes an unbounded bill for responses you never used.

Balance is ground truth

The admin panel shows your real remaining balance, polled from OpenRouter, rather than a figure derived from counting tokens locally.

This is deliberate, and the reasoning is worth stating plainly: local token accounting is exactly what failed before. A system that trusts its own arithmetic about money will be confidently wrong at the worst moment. The number shown is the provider's.

Live activity

Alongside the caps, the panel shows per role:

Reading	Meaning
In flight	Calls executing right now
Calls today	Successful calls since UTC midnight
Tokens today	Tokens billed since UTC midnight
Last call	The most recent completed call, success or failure

One deliberate detail: "last call" does not reset at midnight. Everything else does. If the last call happened at 11:58pm, that stays visible at 12:01am — because the useful question at that moment is "when did this last do anything," and a counter that blanked itself would answer "never."

These are display state only. They are never consulted when deciding whether a call is allowed; the guard reads its own totals.

Key handling

The API key is held by the master and the sidecar, never by the sandbox container. Keys are masked in the admin UI and never written to logs — every diagnostic path redacts them.

That last point is not routine caution. A sidecar's status endpoint is unauthenticated on the local network, so a key appearing in a log line would be a genuine exposure rather than a theoretical one.

What this does not change

Your documents never leave your machine. Search runs against your local index. What goes to OpenRouter is a prompt or a chunk of text for embedding — never a file, never a page image.
Local-only remains fully supported. Every role that could run locally before still can. The default configuration makes no outbound calls.
OCR is local, always. See above.