I scored 27 open-weight model configurations on one frozen 42-task suite. They ran on two machines with different hardware designs and on a metered cloud tier, all under the same judge panel and rubric.
I built the harness to settle a routing decision: which model handles which kind of work, on which machine, before a task ships. A leaderboard screenshot would be out of date the week the next model drops, and most public benchmarks compare models in isolation from the hardware or price tier they’d run on. A routing table needs a different answer: given the machines on my desk and the services I can call over the network, which option delivers the highest quality and still finishes the task? To answer it, I scored local and cloud configurations against the same suite and rubric in the same run.
The Z13 is a Strix Halo APU system with 128GB of unified memory. The P8 is a Lenovo CUDA workstation with Blackwell-generation GPUs and discrete VRAM.
Bionic, from LM Studio, is the path I use for privacy-sensitive work: it sends each job to the Z13 or the P8 and escalates to a zero-data-retention (ZDR) cloud pool only when neither can handle it. That cloud pool is the third tier tested here.
The harness
The suite has 42 tasks across seven weighted domains. Every configuration, local or cloud, runs the same frozen prompts against the same rubric (benchmark v1.2.2).
Judge models score each of the 42 responses against the rubric. A four-model panel scores blind: Gemini 3.5 Flash Lite, GLM-5.3-Flash and Xiaomi MiMo v2.5 as scorers, with DeepSeek V4.1 Flash as tie-breaker. The judges run through OpenRouter, so they sit on infrastructure separate from whatever serves the model under test. Two of them, GLM-5.3-Flash and DeepSeek V4.1 Flash, also appear in the cloud tier below.
A result counts as final, and becomes eligible for cross-model comparison, only at 90% task coverage, 90% completion and at least medium judge confidence. A result that misses any one of the three is marked provisional and excluded from the headline numbers, whatever its raw score.
On this run, 15 of the 26 quality-tested local configurations (9 of 14 on the Z13 and 6 of 12 on the P8), or 58%, ended up provisional, on hardware I fully control.
Routing Setup: Bionic and Hermes
This benchmark covers the two local machines and the Bionic ZDR cloud tier, which are three branches of the routing setup I run day to day. Bionic is the privacy-preserving path. It sends work to the Z13 or P8 directly or escalates to the ZDR cloud tier benchmarked here.
Hermes is the automation and orchestration layer beside it: cron jobs, delegated tasks and a kanban board, dispatching work to the same two local machines or to one of three external providers.
User-initiated work splits into Bionic’s privacy-preserving local path and Hermes’ automation and orchestration path; both resolve to a model provider, local (Z13, P8) or cloud (OpenAI, Anthropic, OpenRouter).
Inside Bionic, every job goes to the Z13 or the P8, whichever has capacity and fits the task, and escalates to the cloud only when neither can handle it, through the ZDR-gated tier. Hermes runs a separate lane for background work that isn’t privacy-sensitive by design, and it can call the external providers directly with no ZDR gate.
The privacy boundary is therefore the entry point: work routed through Bionic stays on local hardware unless the ZDR guarantee is in force, and work routed through Hermes was never treated as sensitive. The numbers below cover the two machines and the ZDR cloud tier.
Controls and data handling
Bionic sessions can’t be called through a direct API, so calibration answers travel through a batch bridge. An extra hop like that can bias a benchmark without anyone noticing, so I ran a control: GPT-OSS-120B through the bridge and directly. The bridge run scored 76.1 and the direct run 78.4, a −2.3-point offset that sits inside the ±3-point tolerance I set beforehand. That result is why I treat the cloud-versus-local comparisons below as measured.
The cloud tier runs entirely through LM Studio’s Bionic pool under its ZDR policy. Pushing 42 tasks’ worth of prompts and model output through a third party raises a data-handling question whenever the content is sensitive. A verified no-retention guarantee made a cloud escalation tier safe to include and kept the benchmark itself from becoming a disclosure risk.
I’m leaving dollar figures out. The harness tracks per-task cost to fractions of a cent, but a specific vendor’s rate card isn’t the point of this piece, and the quality findings stand on their own.
Z13: Strix Halo APU, 128GB unified memory
The Z13 is a Strix Halo APU mini-PC/laptop with 128GB of unified memory and no discrete GPU. Everything shares one memory pool, which changes which model sizes and quantizations are practical to run. Fourteen configurations went through quality testing.
The default I settled on is Qwen3-Coder-30B-A3B. It’s the fastest model on the machine and ranks eighth of the fourteen on quality, with a combined speed-and-quality score of 86.0, which puts it nineteen points ahead of the next entry on that measure. Its weak spot is financial analysis, where it scores 48.0 against a top domain score of 89.5.
The quality ceiling on this machine belongs to Muse-Glimmer-30B at either quantization: 83.9 at Q5_K_M and 83.4 at Q6_K_XL. It’s also the slowest strong-quality option on the fleet, so I use it deliberately where getting the answer right matters more than speed. It wins the financial analysis and research domains and places second on legal/risk assessment.
GLM-4.7-Flash sits at the floor of the tested field on both speed and quality, so skip it for these use cases.
The Muse-Glimmer quantizations run against what bit width alone would predict. Q5_K_M, the smaller-footprint quant, beats Q6_K_XL by 1.9 points on financial analysis. The gap between them is only 0.5 points overall, so the domain score shows a difference the aggregate scores dilute.
14 locally-hosted configurations × 7 benchmark domains · judge-scored, 0–100 · ★ = top score in that domain · sorted by overall quality
P8: CUDA workstation, Blackwell-generation GPUs
The P8 is a CUDA workstation with Blackwell-generation GPUs and discrete VRAM, so it has no unified-memory ceiling and suits larger active-parameter counts. Twelve configurations went through quality testing.
Qwen3.5–122B-A10B posts the best quality score among eligible configurations, 84.6, and the best long-context score in the entire set, 96.2. It’s a mixture-of-experts model with roughly 10B active parameters, and the scores justify its size.
For latency-sensitive work, Qwen3-Coder-Next is the faster eligible default, with 100% completion and the strongest research/evidence score on the machine.
Qwen3-VL-235B-A22B (Instruct) posts the standout figure of the whole assessment, 95.4 on document fidelity, but its throughput collapses at long context, so I treat it as a document-review specialist.
Qwen3.5–397B-A17B (Thinking) and Qwen3.8–27B (Instruct) are flagged do-not-deploy-as-configured. The first truncated on 47.6% of tasks and returned blank output on most of those. The second posted the lowest quality score on the machine, 46.3. Both failures surface only when you run the full suite instead of spot-checking a handful of prompts.
Qwen3.5–397B-A17B (Instruct) and Qwen3.5–122B-A10B changed completion behavior between passes, from roughly 58–60% to 98–100%, with no change on my end. I haven’t yet traced the serving-configuration difference behind the jump.
The P8 quality run also swapped three of the four OpenRouter judges from the prior pass. That changes the panel itself, so I’m treating this run as a rebaseline, and I don’t claim clean comparability with earlier P8 numbers.
12 locally-hosted configurations × 7 benchmark domains · judge-scored, 0–100 · ★ = top score in that domain · sorted by overall quality
Side by side, the two machines show the speed-quality trade-off. The Z13 clusters toward the low-speed, high-quality corner whenever it isn’t running its fast default, while the P8 spreads more evenly across the speed axis. Bubble size adds a third axis, the quantized model footprint, and parameter count and quality don’t move together on either machine: some of the largest bubbles sit mid-field, and some of the smallest hold their own against configurations many times their size.
12 configs per machine · bubble area = total parameters (quantized footprint) · Speed Index is field-relative, not cross-machine absolute · judge-scored quality, 0–100
The cloud tier: quality clusters, cost doesn’t
Four Bionic ZDR cloud configurations cleared calibration, all final and all at 100% completion. Quality varies by less than 3.5 points across the four, while cost varies roughly 21x.
The cheapest, GLM-5.3-Flash, scores 92.77. The most expensive, Kimi K3, scores 92.46 at over twenty times the run cost. DeepSeek V4.1 Flash takes the top overall score, 93.00, and is the second-cheapest. These scores apply to my 42 business tasks, do your mileage may vary.
The same pattern appears inside one model family, where price and lineage should track most closely. GLM-5.3 and GLM-5.3-Flash share an architecture, and GLM-5.3 costs roughly nine and a half times more to run the identical battery. Its overall quality is three points below its cheaper sibling’s. On document fidelity, which covers extraction and structuring work, it falls to 77.4 against 92.6 for the Flash model it’s meant to supersede, a 15-point gap in the wrong direction.
Routing by vendor tier fails here: the larger, pricier sibling isn’t a strict upgrade, and an average across domains hides where it loses.
4 calibrated cloud configurations × 7 benchmark domains · judge-scored, 0–100 · ★ = top score in that domain · sorted by overall quality
If you’re building your own harness
- Domain scores. Report them next to the overall score. The two Muse-Glimmer quantizations sit 0.5 points apart overall and 1.9 apart on financial analysis, and GLM-5.3 trails its Flash sibling by 3 points overall and by 15 on document fidelity.
- Set the eligibility bar before you see results. The three gates only work if a configuration at 89% completion isn’t rounded up to 90% because the number is inconvenient. On this run, 15 of 26 local configurations stayed provisional.
- Any bridge needs a control run. A proxy or batch transport between the harness and the model can bias results without showing up in the scores. Rerun a model you already trust through the same path and compare the delta against a tolerance you set in advance; mine was −2.3 points against ±3.
- Price and quality. Measure them separately. The cloud tier’s most expensive model, at over twenty times the cost of its cheapest, scored below it, and the top scorer was the second-cheapest.
- A changed judge panel starts a new baseline. When the panel differs between runs, treat the new run as a rebaseline even if the harness and suite are otherwise identical, and don’t average across the swap.
What’s next
Two items remain open: the serving-configuration difference behind the P8 completion jump, and comparability between the rebaselined P8 run and earlier P8 numbers. Cloud pricing resets monthly, so the cost findings will date first.
None of this is a final verdict on any of these models. I’ll rerun the suite when enough of the roster turns over.
This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!
I Benchmarked 27 LLM Configurations Across Two Machines and a Zero-Retention Cloud Tier was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.
By Ivan Ruzic, Ph.D. · Originally published on Medium