The question we wanted answered: can you turn a small, quiet mini PC into a real LLM serving box by hanging a GPU off it through a Thunderbolt enclosure — or does the cable quietly ruin everything? So we measured it, both ways, on the same card.
An Intel Arc Pro B70 (32 GB) in a Thunderbolt/USB‑4 enclosure, driven from a small AMD Ryzen mini PC over a single USB‑4 40 Gb/s cable. Inside the enclosure the card negotiated a PCIe 4.0 ×4 link. We ran the same 30B-class model (a hybrid‑MoE, ~24 GB of weights) against the exact same B70 sitting in an ordinary internal PCIe slot on a desktop — same driver, same runtime, same model file. Only the connection changed.
The mental model going in: a GPU is a coprocessor at the far end of a bus. Move it from a full x16 slot to a x4 link over a cable and you cut bus bandwidth roughly eightfold. The interesting question is whether inference actually cares.
| Internal PCIe slot | USB4 eGPU (40 Gb/s) | Difference | |
|---|---|---|---|
| Warm decode (30B model) | 46.3 / 45.5 / 44.7 tok/s | 40.1 / 40.9 / 41.6 tok/s | ≈10% slower |
| Cold model load (~24 GB, caches dropped) | 11.1 s | 33.4 s | 3× slower |
| Prompt processing | within noise | within noise | ≈0 |
Three consecutive warm runs each side; the model resident in VRAM, one request at a time. The pattern is the whole story: token generation shrugged off the cable, and loading the model paid dearly for it.
Autoregressive decoding is memory-bandwidth-bound, not bus-bound. Once the ~24 GB of weights are sitting in the GPU's own VRAM, generating each token means streaming those weights through the GPU's memory system once — tens of terabytes per second of on-card bandwidth that the USB‑4 cable never touches. What actually crosses the cable per token is tiny: a few kilobytes of activations and the new token id. USB‑4 gives you roughly 3 GB/s of usable bandwidth here, and decode simply doesn't ask for it.
So where did the ~10% go? Not bandwidth — latency. Every token involves a handful of small host↔GPU round-trips (kick off the kernel, sync, hand back the result), and each of those round-trips is a little slower across a Thunderbolt bridge than across a motherboard trace. Ten percent is the accumulated cost of those round-trips, not of moving data. It's the kind of penalty that shrinks further with batching and longer generations, because the fixed per-step overhead gets amortized over more work.
Cold-loading the model took three times longer over USB‑4, and that number surprised us until we found the cause. Loading streams the full 24 GB from disk → system RAM → across the cable → into VRAM — a genuinely bandwidth-heavy operation, the one part of the lifecycle that does lean on the bus. But 3× is worse than an eightfold-narrower link alone would explain, and the reason is a detail deep in how the enclosure exposes the card.
On this rig the card came up with only a 256 MB CPU-visible memory window — no Resizable BAR. The mini PC's firmware had no Above-4G decoding or ReBAR options to enable, and a kernel-level attempt to grow the window returned -ENOSPC: the USB‑4 bridge apportions its address windows at boot and won't budge afterward. So all 24 GB of weights have to be shoveled into VRAM through a 256 MB porthole, one small window at a time, instead of mapped in big contiguous strides. That's the missing multiplier.
The small memory window has a nasty side effect worth its own paragraph, because it cost us an afternoon. Intel's OpenCL runtime refuses to initialize at all on a small-BAR device — clinfo reports zero platforms, as if no Intel GPU is present. Vulkan, meanwhile, works perfectly and serves the model fine. So any tool that detects GPUs by asking OpenCL — and plenty do — will conclude the card doesn't exist, while a Vulkan path runs happily on the same hardware. If you're building GPU detection, don't trust OpenCL alone: fall back to Vulkan, or you'll declare a working 32 GB card missing.
An eGPU is a legitimate way to build an LLM serving box. You keep essentially all of the decode speed (~90% of internal-slot throughput), you get the card's full 32 GB of VRAM to hold real models, and you turn a silent little mini PC into a GPU server with one cable. The trade-offs are honest and small: a ~10% decode tax, a one-time slow cold load, and a driver-detection quirk you route around once.
If you're speccing a brand-new build, put the card in a real slot — that 10% is free performance and the cold loads stay fast. But if the machine you already own is small and the GPU you have is external, the answer is no longer "don't bother." It works, and now you know exactly what it costs.
This is not an academic exercise for us. Ntersection is a gateway that routes AI inference to a fleet of GPUs wherever they physically live — a desktop, a lab, a mini PC in a closet, a card on a cable behind a residential NAT. Each machine dials out to the gateway over one held-open connection and serves requests backward through it, so there are no inbound ports to open and no bespoke networking per box. To the gateway, this eGPU rig is just another node advertising 32 GB and a model — it neither knows nor cares that the card is at the end of a Thunderbolt cable. That's the point: capable GPUs are scattered everywhere, and the interesting work is making all of them serve, exactly as they are.
Numbers here are from our own lab, same methodology as the GPU benchmarks page. Want to measure your own card? Nbench is a free single binary that runs the same tests against your local runtime.