GPU Sidecar: Your Private AI Engine
Run AI models on your own hardware — Mac, Windows, or Linux. No data leaves your network, ever. One master coordinates a whole fleet of GPU machines for full document intelligence, completely local.
Why Local AI Matters for Legal Work
When you use a cloud AI service, your document text is transmitted to a third party for processing. For legal work — where attorney-client privilege and confidentiality are paramount — that creates risk.
The GPU Sidecar eliminates this concern entirely. Every AI operation — reading scanned documents, generating search indexes, ranking results, answering questions — happens on hardware you control. Nothing is transmitted to OpenAI, Anthropic, or any other provider.
Without the Sidecar
Sound Suite still works perfectly with cloud API keys. You add an OpenAI or Anthropic key in the admin panel, and your documents are processed through their services. The quality is excellent — but your document text is sent to those providers.
Where Your Data Goes
What the Sidecar Does
Four AI capabilities that run entirely on your hardware. No cloud services, no API costs, no data leaving your network.
Reading Scanned Documents (OCR)
Many court filings arrive as scanned PDFs — just images with no searchable text. The sidecar runs a specialized AI model (olmocr2) that reads these scans with high accuracy, even on poor-quality copies or faded documents.
Making Documents Searchable by Meaning
Think of embedding as creating a meaning fingerprint for each paragraph — your private index that understands legal concepts, not just keywords. This is how Sound Suite finds relevant passages when you search "what obligations does the contract impose" — it understands meaning.
Getting the Best Results (Reranking)
After vector search returns ~100 candidate passages, a second AI model (a cross-encoder) re-reads the query alongside each result and re-scores them. Think of it as having a senior associate review your search results and put the most relevant ones first. Takes 1-2 seconds for 100 documents. Auto-starts on demand and idles after 5 minutes to save GPU memory.
AI Answers and Auto-Suggest
Powers the AI Chat panel and Auto-Suggest in the draft editor. The AI reads your search results and formulates answers with citations — or suggests the next sentence as you write, drawing from your indexed case documents.
How It Works
The sidecar is a lightweight agent that runs on any machine with a GPU — an Apple Silicon Mac, a Windows PC, or a Linux server. Each sidecar dials out to your Sound Suite master over a gossip protocol, so a whole fleet of GPU machines can work together behind firewalls, NAT, or on separate subnets — with nothing to port-forward.
Setup in Four Steps
- 1 Download the sidecar — It ships as a standalone package that runs on Apple Silicon (Metal), Windows (Docker WSL2 + CUDA), or Linux (Docker + NVIDIA).
- 2 Auto-provisioning — The sidecar detects its OS and sets itself up: on Windows and Linux it pulls a Docker image per model; on a Mac it talks to native Ollama, which has the GPU access Docker can’t get on Apple Silicon. No manual configuration needed.
- 3 Join the fleet — Point the sidecar at your master. It opens an outbound gossip connection (a WebSocket tunnel, with HTTP fallback) and reports its OS, GPU, and live status every few seconds.
- 4 Assign roles from the master — From one admin screen, choose which machine runs each AI model. The master only offers combinations that machine’s OS can actually run, then broadcasts the plan to every sidecar.
Model Roles
Embedding
ollama · qwen3-embedding:0.6b
Completion
ollama · qwen3.5:9b
OCR
ollama · olmocr2:7b-q8
Reranker
vLLM · Qwen3-Reranker-8B
OS-Aware Role Assignment
The master knows each machine’s operating system and only assigns models it can actually run. Spread roles across the fleet — embedding on a Mac, the reranker on a Windows NVIDIA rig, completion on a Linux server.
| Role | Mac | Win | Linux |
|---|---|---|---|
| Embedding | ✓ | ✓ | ✓ |
| Completion | ✓ | ✓ | ✓ |
| OCR | ✓ | ✓ | ✓ |
| Reranker | — | ✓ | ✓ |
On Apple Silicon, models run on native Ollama (Metal); Windows uses Docker WSL2 (CUDA). VRAM is budgeted per machine — no single 24 GB card required to do everything.
Gossip Protocol
Master
Sound Suite Server
Sidecar Fleet
Mac · Windows · Linux
Sidecars dial out to the master — nothing to port-forward. The master broadcasts model assignments down the tunnel; each sidecar streams status back. Works behind NAT and firewalls.
With vs Without the Sidecar
Sound Suite works great either way. The sidecar adds speed, privacy, and eliminates API costs.
| Capability | Without Sidecar | With Sidecar |
|---|---|---|
| Document search | Cloud API or slow local CPU | Fast local GPU |
| OCR accuracy | Good (tesseract.js) | Excellent (AI-powered) |
| Search quality | Basic vector search | Reranked results |
| AI chat / auto-suggest | Requires API keys ($) | Included, no API costs |
| Data privacy | Text sent to cloud providers | 100% local |
| Speed | Depends on internet / CPU | GPU-accelerated |
Don't Have a GPU? You Don't Need One
Sound Suite works perfectly without the sidecar. Add API keys for OpenAI, Anthropic, or Groq in the admin panel and you get excellent AI capabilities immediately. No hardware investment, no Docker setup.
Cloud API Mode
- ✓ No hardware requirements
- ✓ Excellent AI quality from top providers
- ✓ Easy setup — just add an API key
- • Document text processed by third parties
- • Ongoing API costs
GPU Sidecar Mode
- ✓ 100% local — nothing leaves your network
- ✓ No API costs after setup
- ✓ GPU-accelerated speed
- ✓ AI reranking for better search results
- ✓ Runs on Mac, Windows, or Linux — spread across a fleet
- • Needs a 12+ GB GPU (Apple Silicon, or NVIDIA on Windows/Linux)
What You Need
Three requirements. The sidecar handles everything else.
A GPU — Any OS
12+ GB of GPU memory: an Apple Silicon Mac, or an NVIDIA card (RTX 3060 or better) on Windows or Linux. Mix and match across a fleet.
Docker or Ollama
Docker Desktop (Windows WSL2 / Mac) or Docker Engine + NVIDIA Container Toolkit (Linux). On Apple Silicon, the sidecar drives native Ollama for GPU access. Setup is automatic.
Outbound Network
Each GPU machine only needs to reach your master — it dials out over the gossip protocol. No inbound ports, no port-forwarding. Same LAN, VPN, or across the internet.
Ready to Go Fully Local?
Download Sound Suite and add the GPU Sidecar for complete on-premise AI.
Sidecar license: $1,000 per instance for commercial use. Free for pro se litigants.