GPU Benchmarks

Mobile: a flagship phone, benchmarked honestly

Same tool, same methodology, no asterisks: we ran the site’s stock nbench arm64 binary on a Samsung Galaxy Z Fold8 Ultra (Snapdragon 8 Elite Gen 5, 12GB LPDDR5X ≈85GB/s) inside Termux, against Ollama running on the phone itself. CPU inference — phones don’t expose their GPUs to this stack — plugged in, wake-lock held.

What the phone teaches

• 3–4B models are genuinely conversational on a 2026 flagship — 13–17 tok/s is faster than most people read.
• The ceiling case: a 6.6GB model in a 12GB phone. Ornith-1.5:9b loads with nothing to spare and decodes a rock-steady 6.4 tok/s — but prompt processing thrashes at the memory ceiling. The largest class that fits is not the largest class that’s pleasant: 2–4GB models are the phone sweet spot.
• Different models throttle differently: llama3.2 held ~16 sustained, but gemma3:4b fell from a 16.2 cool burst to ~13 within a single 3-run round — heavier per-token compute finds the thermal ceiling faster. Cool-vs-sustained is the honest pair to report on mobile.
• Thermal throttling is asymmetric, and it proves the site’s thesis: decode dropped only ~5% hot (memory-bandwidth-bound) while prompt processing fell 34% (compute-bound). Heat punishes compute; bandwidth barely notices.
• The RAM spec is marketing; the app allowance is real. A “12GB” phone gives an app 5–6GB before the OS starts killing — size models to that, not the box label.
• Verdict vs the fleet: the phone is ~1/27th of an RTX 5090 on the same model — and it fits in a pocket and sips 10W. Both facts belong on the same page.

Battery: what we learned before the phone left

• Screen off + unplugged = AI is effectively impossible: 0.27 tok/s. Samsung’s background execution limits collapse inference by ~60× the moment the screen sleeps on battery (watchable in real time: 1.7 → 0.27 tok/s as the throttle deepens). Plugged-in background work is exempt. The implication for anyone dreaming of phones as always-on AI workers: only while charging.
• There is NO battery throttle with the screen on. Unplugged screen-on decode measured 17.0 tok/s — identical to plugged (17.01). The battery isn’t the limiter; the screen state is everything.

Methodology: Termux + pkg install ollama, the standard nbench-linux-arm64 download from this site, standard warm-run methodology, three back-to-back rounds for the thermal curve. Phone entries appear on the leaderboard alongside everything else.

Ornith-1.5: first independent throughput numbers

Ornith-1.5 is the brand-new open-model family from DeepReinforce’s Ornith team — MIT-licensed, released this month, on Ollama in three sizes: 9B dense (6.6GB), 35B mixture-of-experts (23GB, ~3B active parameters per token), and a 397B MoE (242GB). All variants carry a 256K context window. Its claim to fame is how it was trained: the model generated a meaningful part of its own curriculum — proposing tasks, building the harnesses that graded them, and producing the solutions that fed its reinforcement learning. The pitch is a shift from “scale up data and compute” to “scale up the model’s ability to improve itself.”

What it’s good at

Its sweet spot is coding and agentic work — terminal and tool use, multi-step software tasks, running as the brain of a coding agent — and it plugs into the usual local ecosystem (Ollama, llama.cpp, agent frameworks). The vendor’s benchmarks put it trading blows with frontier closed models on agentic coding — slightly ahead of Claude Opus 4.8 on Terminal-Bench 2.1 (86.1 vs 85), slightly behind on DeepSWE — remarkable territory for something you run locally for free. The 35B MoE reportedly beats comparable open dense models (Qwen 3.6-35B, Gemma 4-31B) while activating only ~3B parameters per token, so it is much cheaper to run per response than its size suggests. And the standard local advantages apply: no per-token cost, full privacy, no rate limits, quantized formats from day one.

Caveats before you get excited

Every benchmark number above is vendor-self-reported with no independent verification yet — it launched days ago, early write-ups already show small score discrepancies, sources conflict on whether image input actually works, and the self-curriculum approach is novel and unproven outside its own benchmarks. Which is exactly why this section exists: below are independent, reproducible throughput measurements on real hardware, same methodology as everything else on this page. A proper coding-quality evaluation is planned separately.

These numbers measure speed, not smarts — how fast it runs where you’d actually run it.

Notes from the runs

• The finding of the batch: on bandwidth-starved cards, the 35B MoE is FASTER than the 9B dense. On both B70s the 35B outruns the 9B (42.5 vs 32.5 tok/s) — ~3B active parameters touch fewer bytes per token than a 9B dense pass. “Runs like a 3B, knows like a 35B” measured independently: an affordable Intel card serves the flagship variant at a conversational 43 tok/s. On the 5090’s huge bandwidth the ordering flips back (212 vs 183) — small dense wins when bandwidth is free.
• The eGPU story holds a third time: USB4 enclosure vs internal slot is a wash on the MoE (43.1 vs 42.5 — within noise) and ~3% on the 9B.
• It runs on a phone. The 9B loads on a Galaxy Z Fold8 Ultra (12GB) with nothing to spare and decodes a steady 6.4 tok/s — but prompt processing thrashes at the memory ceiling. Proof-of-possible, not daily-drivable; the vendor’s ~1.5GB mobile build (not yet on Ollama) is the right tool for that job.
• The 35B MoE’s economics check out: ~3B active parameters means it decodes far faster than a 35B dense model would — the MoE trick our fleet’s own 30B model relies on, working as advertised here.
• The 4090 sits this one out at 35B — 23GB of weights plus cache doesn’t fit 24GB. The 9B runs everywhere.

One model across two machines: two GeForce cards plus a DGX Spark, over the house LAN

A 56GB pair of cards (RTX 5090 32GB + RTX 4090 24GB) runs a 60GB model only by spilling part of it into host RAM, and a 153GB model barely at all. Sitting on the same LAN was an ASUS Ascent GX10 — a DGX Spark, GB10, 128GB of unified memory of which 124GB is GPU-addressable. llama.cpp’s RPC backend lets one llama-server or llama-bench place layers, or individual tensors, on a ggml-rpc-server running on another machine, over plain TCP. We measured what that is worth with nothing bought: first through the house switch at 5 GbE on the head’s side, then with a USB4 10 GbE adapter on the head so both ends ran at 10 GbE through the same switch.

Head: AMD Ryzen 9 7950X3D, 250GB DDR5 at 3600 MT/s, RTX 5090 (PCIe 5.0 x8) + RTX 4090 (PCIe 4.0 x8), CUDA 13.0. Worker: GX10, 10 GbE RJ45. Same llama.cpp commit (50a6c5c) on both ends — the RPC protocol requires it.

What the numbers say

• The 60GB model got 1.8× faster by leaving the box. Not because the Spark is fast — its 273 GB/s is a sixth of the 5090’s — but because its memory is all GPU-addressable, and the alternative was host RAM at 3600 MT/s. The ladder is: 5090 → 4090 → Spark → RAM. Weights go on the fastest device with room, and what every token touches (the dense part) goes highest.
• Mixture-of-experts models reward placing tensors, not layers. Grok-2’s dense 43GB fits the two cards; its 110GB of experts went to the Spark with one --override-tensor rule. That beat the best plain layer split by 24% on decode — and it is what makes a 153GB model usable at all on 56GB of VRAM.
• Bandwidth was never the limit; latency is. Going from 5 to 10 GbE on the head’s side changed decode by nothing (116.8 → 116.0; 4.71 → 4.74) and prompt processing by +11% and +33%. Per generated token only kilobytes cross the wire, but in the expert layout they cross it 128 times per token at roughly 0.3 ms a hop. Where Grok-2’s 212 ms per token goes: ~135 ms the Spark reading experts, ~36 ms the cards reading dense weights, ~40 ms hops and RPC overhead. A direct cable between the machines would not change that: a switch hop is microseconds. Both boxes stayed on the switch.
• Loading is the only time the link matters: the worker caches every tensor it receives on its own NVMe, so a model crosses the network once. Our second load of Grok-2 came from the Spark’s cache at NVMe speed.

Adding the eGPU: capacity, not speed

We also put the Arc Pro B70 32GB from the enclosure tests above on the head’s USB4 port as a second RPC worker, on a Vulkan build of the same commit (so the worker exposes only the Intel card). It works, and it is honest about what Vulkan on Battlemage costs:

Through Mesa’s Vulkan driver the B70 moves about 1.2 ms per gpt-oss layer: the same cost as this head’s host RAM, a tenth of the 5090, and slower per expert layer than the Spark. So in this pool it adds 32GB of room for models that would otherwise not fit, and nothing to speed. The 35.7 tok/s on gpt-oss-20b (3.6B active parameters) sits right where the Ornith 35B MoE (~3B active) landed on the same card under Ollama — it is the card’s Vulkan ceiling, not an RPC tax. Our own bake-off above puts SYCL over OpenCL at +17% on this silicon; that is worth taking, and it will not change the ranking.

Traps, so you do not hit them

• --tensor-split lists the remote devices first (RPC0 / CUDA0 / CUDA1), the opposite of what --list-devices prints. With two workers the order is RPC0 / RPC1 / CUDA0 / CUDA1.
• In --override-tensor the remote buffer type is named RPC0[host:port], and the index counts per server: a second worker’s only device is also RPC0[…]. In llama-bench put --rpc before -ot or the name is “unrecognized”; and separate several rules with ; there — a comma starts a second benchmark configuration and the later rules silently vanish from the first.
• -ncmoe N sends the first N layers’ experts to RAM, and those layers sit on the first local card; the default VRAM-proportional split then overflows the other card. Give -ts by hand.
• Both ends on the same llama.cpp commit; the worker is unauthenticated, so keep it on a LAN you trust.
• A USB4 10 GbE adapter (Aquantia AQC113) on the same host controller as a powered eGPU: when the enclosure powers on, the whole USB4 tree re-enumerates and the adapter’s bridge can come back with a 3MB memory window for a device that needs 4.7MB — the driver fails with error -5. Removing that bridge from the PCI tree and rescanning its parent port fixes it live; powering the eGPU on before plugging the adapter avoids it.

Reproduce: on the worker ggml-rpc-server -H 0.0.0.0 -p 50052 -c; on the head llama-bench --rpc worker:50052 -m gpt-oss-120b-MXFP4.gguf -ngl 99 -ts 10/10/16 -p 512 -n 128, and for Grok-2 -ts 0/26/38 -ot 'blk\..*\.ffn_(up|down|gate)_exps\.weight=RPC0[worker:50052]'. GGUF files from Hugging Face (ggml-org/gpt-oss-120b-GGUF, unsloth/grok-2-GGUF Q4_K_M).

Methodology

• Serving (warm) numbers: Ollama /api/generate, non-streamed; tok/s = eval_count / eval_duration as reported by the runtime. Model resident in VRAM, 3 consecutive runs of the same prompt, all runs shown or averaged as labeled.
• Cold load: systemctl restart ollama, then echo 3 > /proc/sys/vm/drop_caches (so the OS page cache can’t hide the storage path), then one 5-token generation; load time = load_duration.
• Backend bake-off: llama-bench -p 512 -n 128 -r 3 -ngl 99 on llama.cpp build 3f545be, pointed at the identical GGUF file for every backend. ± figures are llama-bench’s own run variance.
• NVFP4 rows: vLLM (pip venv install on x86 + CUDA), model NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, GPU pinned via CUDA_VISIBLE_DEVICES; decode measured client-side over streamed chunks, first token to last, 600-token runs.
• Small-model ladder: each GPU isolated with a dedicated Ollama instance (CUDA_VISIBLE_DEVICES pin, 8k context so multi-GPU boxes don’t balloon the KV cache across cards), llama3.2:3b, standard 3-run methodology.
• Parallel test: two threads POSTing simultaneously, wall-clock and per-stream time-to-first-token recorded client-side.
• Environment: Ubuntu 24.04.4, Linux kernel 7.0, Ollama 0.32.15, Mesa 25.2.8 (Vulkan), Intel compute-runtime (NEO) 26.27 + IGC 2.38.2 (OpenCL), oneAPI DPC++ 2026.1 (SYCL build). DGX Spark: NGC vLLM 26.07-py3 container, NVFP4 model build. eGPU link: USB4 40Gb/s, enclosure slot negotiated PCIe 4.0 x4.
• Prompt note: Nbench v0.3+ uses a shorter warm-up prompt than the earliest runs on this page; its numbers can read a few percent higher than the tables here. Leaderboard entries are all same-prompt, so they compare cleanly with each other.
• Two-machine pool: llama.cpp commit 50a6c5c (CUDA 13.0 on the head, the RPC backend, a Vulkan build of the same commit for the Intel worker), llama-bench -p 512 -n 128 -r 1 -ngl 99 with every placement given explicitly; single runs, so treat differences under ~3% as noise. Link throughput measured with a 4GB single-stream TCP transfer each way.
• Honesty notes: single-request figures unless stated; multi-stream serving divides these numbers across streams. Your results will vary with quantization, context length, and prompt shape — compare like with like.

Run this on your own GPU: Nbench (free)

Nbench is our benchmark as a single free binary. Run it plain for the CLI — or run nbench -ui for a browser panel that shows your detected hardware, lists every model your Ollama already has, streams the run live, and shows the model’s actual answer alongside the number. It — no install, no dependencies, no accounts, and it never phones home. It benchmarks your local Ollama with the exact methodology above (hardware detection, warm decode, model load, and the 2-stream parallel check), so the number it prints is directly comparable to every number on this page. Windows, Linux, and macOS.

The installer detects your system, puts nbench on your PATH (Windows also gets a Start Menu shortcut), and never needs administrator rights. Paste it into a terminal (PowerShell on Windows).

Installing or downloading requires accepting the as-is/no-warranty terms (one click). Needs a local Ollama. Default model is llama3.2:3b (a ~2GB one-time pull) so anyone can run it; to compare directly against the table above, run -model nemotron-3.5-lightning:latest (a ~25GB pull — needs a card with roughly 26GB+ of free VRAM); benchmark any other model with -model <name>, and -json - emits the full machine-readable result. A community leaderboard — opt-in submission of your score — is coming next.