GPU Sidecar: Your Private AI Engine
Run AI models on your own hardware — Mac, Windows, or Linux. No data leaves your network, ever. One master coordinates a whole fleet of GPU machines for full document intelligence, completely local.
Why Local AI Matters for Legal Work
When you use a cloud AI service, your document text is transmitted to a third party for processing. For legal work — where attorney-client privilege and confidentiality are paramount — that creates risk.
The GPU Sidecar eliminates this concern entirely. Every AI operation — reading scanned documents, generating search indexes, ranking results, answering questions — happens on hardware you control. Nothing is transmitted to OpenAI, Anthropic, or any other provider.
Without the Sidecar
Sound Suite still works perfectly with cloud API keys. You add an OpenAI or Anthropic key in the admin panel, and your documents are processed through their services. The quality is excellent — but your document text is sent to those providers.
Where Your Data Goes
What the Sidecar Does
Four AI capabilities that run entirely on your hardware. No cloud services, no API costs, no data leaving your network.
Reading Scanned Documents (OCR)
Many court filings arrive as scanned PDFs — just images with no searchable text. The sidecar runs a specialized AI model (olmocr2) that reads these scans with high accuracy, even on poor-quality copies or faded documents.
Making Documents Searchable by Meaning
Think of embedding as creating a meaning fingerprint for each paragraph — your private index that understands legal concepts, not just keywords. This is how Sound Suite finds relevant passages when you search "what obligations does the contract impose" — it understands meaning.
Getting the Best Results (Reranking)
After vector search returns ~100 candidate passages, a second AI model (a cross-encoder) re-reads the query alongside each result and re-scores them. Think of it as having a senior associate review your search results and put the most relevant ones first. Takes 1-2 seconds for 100 documents. Auto-starts on demand and idles after 5 minutes to save GPU memory.
AI Answers and Auto-Suggest
Powers the AI Chat panel and Auto-Suggest in the draft editor. The AI reads your search results and formulates answers with citations — or suggests the next sentence as you write, drawing from your indexed case documents.
How It Works
The sidecar is a lightweight agent that runs on any machine with a GPU — an Apple Silicon Mac, a Windows PC, or a Linux server. Each sidecar dials out to your Sound Suite master over a gossip protocol, so a whole fleet of GPU machines can work together behind firewalls, NAT, or on separate subnets — with nothing to port-forward.
Setup in Four Steps
- 1 Download the sidecar — It ships as a standalone package that runs on Apple Silicon (Metal), Windows (Docker WSL2 + CUDA), or Linux (Docker + NVIDIA).
- 2 Auto-provisioning — The sidecar detects its OS and sets itself up: on Windows and Linux it pulls a Docker image per model; on a Mac it talks to native Ollama, which has the GPU access Docker can’t get on Apple Silicon. No manual configuration needed.
- 3 Join the fleet — Point the sidecar at your master. It opens an outbound gossip connection (a WebSocket tunnel, with HTTP fallback) and reports its OS, GPU, and live status every few seconds.
- 4 Assign roles from the master — From one admin screen, choose which machine runs each AI model. The master only offers combinations that machine’s OS can actually run, then broadcasts the plan to every sidecar.
Model Roles
Embedding
ollama · qwen3-embedding:0.6b
Completion
ollama · qwen3.5:9b
OCR
ollama · olmocr2:7b-q8
Reranker
vLLM · Qwen3-Reranker-8B
OS-Aware Role Assignment
The master knows each machine’s operating system and only assigns models it can actually run. Spread roles across the fleet — embedding on a Mac, the reranker on a Windows NVIDIA rig, completion on a Linux server.
| Role | Mac | Win | Linux |
|---|---|---|---|
| Embedding | ✓ | ✓ | ✓ |
| Completion | ✓ | ✓ | ✓ |
| OCR | ✓ | ✓ | ✓ |
| Reranker | ● | ✓ | ✓ |
| Deep Search (RLM) | ● | ✓ | ✓ |
✓ runs locally · ● runs without a local GPU, via the OpenRouter path.
On Apple Silicon, models run on native Ollama (Metal) or Docker Model Runner; Windows uses Docker WSL2 (CUDA). VRAM is budgeted per machine — no single 24 GB card required to do everything.
Gossip Protocol
Master
Sound Suite Server
Sidecar Fleet
Mac · Windows · Linux
Sidecars dial out to the master — nothing to port-forward. The master broadcasts model assignments down the tunnel; each sidecar streams status back. Works behind NAT and firewalls.
With vs Without the Sidecar
Sound Suite works great either way. The sidecar adds speed, privacy, and eliminates API costs.
| Capability | Without Sidecar | With Sidecar |
|---|---|---|
| Document search | Cloud API or slow local CPU | Fast local GPU |
| OCR accuracy | Good (tesseract.js) | Excellent (AI-powered) |
| Search quality | Basic vector search | Reranked results |
| AI chat / auto-suggest | Requires API keys ($) | Included, no API costs |
| Data privacy | Text sent to cloud providers | 100% local |
| Speed | Depends on internet / CPU | GPU-accelerated |
Don't Have a GPU? You Don't Need One
Every inference role can now run through OpenRouter — embeddings, reranking, completions and Deep Search. Pick the mode per role, and change your mind later without reindexing.
OCR is the one exception, and that is deliberate. It always runs locally, so the page images of your filed documents are never transmitted anywhere — under any configuration. Why →
Cloud API Mode
No sidecar at all
- ✓ No hardware, no Docker
- ✓ Add a provider key and go
- • Document text processed by third parties
- • Ongoing API costs
OpenRouter Mode
Sidecar, no GPU needed
- ✓ Embedding, reranking, completion and Deep Search all routed
- ✓ Deep Search runs in a CPU-only container — no CUDA, no weights
- ✓ OCR still local — page images never leave
- ✓ Per-role choice; swap without reindexing
- • Chunk text is sent for embedding; metered API costs
Full Local Mode
Sidecar with a GPU
- ✓ 100% local — nothing leaves your network
- ✓ No API costs after setup
- ✓ GPU-accelerated throughput
- ✓ Mac, Windows or Linux — spread across a fleet
- • Needs a 12+ GB GPU
Every role defaults to local. Nothing is routed outward until you configure it.
What You Need
Three requirements. The sidecar handles everything else.
A GPU — Optional
For fully-local inference: 12+ GB of GPU memory — an Apple Silicon Mac, or an NVIDIA card (RTX 3060 or better) on Windows or Linux. Mix and match across a fleet.
Without one, route roles through OpenRouter and the sidecar still runs — on CPU.
Docker or Ollama
Docker Desktop (Windows WSL2 / Mac) or Docker Engine + NVIDIA Container Toolkit (Linux). On Apple Silicon, the sidecar drives native Ollama for GPU access. Setup is automatic.
Outbound Network
Each GPU machine only needs to reach your master — it dials out over the gossip protocol. No inbound ports, no port-forwarding. Same LAN, VPN, or across the internet.
Ready to Go Fully Local?
Download Sound Suite and add the GPU Sidecar for complete on-premise AI.
Sidecar license: $1,000 per instance for commercial use. Free for pro se litigants.