PrismML Bonsai 2 27B - A 27B Model in 5.9GB at 98.2% Retention, and What 1.76 Bits-Per-Weight Does to the Local-AI Deployment Envelope

PrismML shipped a 27.8B-parameter model in 5.9GB at 98.2% of Qwen3.8 27B's benchmark aggregate. Packed ternary weights, FP16 group scaling, 0.714 mWh/token on RTX 4090. Training compute unchanged; inference memory collapsed.

September 18, 2026 - PrismML shipped Ternary Bonsai 2 27B on September 17: 27.8B parameters stored at 1.76 effective bits per weight, 5.9GB on disk, 98.2% of Qwen3.8 27B’s 20-benchmark aggregate score, 0.714 mWh/token on an RTX 4090, up to 143 tok/s on an RTX 5090. The model is full end-to-end ternary {-1, 0, +1} with FP16 group-wise scaling - packed weights are consumed directly by the runtime, never expanded back to FP16. The structural argument is not “BitNet is back.” It is “the deployment envelope just collapsed by 9x and the benchmark gap did not.”

The retention table: Bonsai 2 vs Qwen3.8 27B, 20-benchmark aggregate 83.9 vs 85.4

PrismML’s own 20-benchmark suite breaks down per row. Qwen3.8 27B is the FP16 base model Bonsai 2 was quantized from; the comparison is apples-to-apples on the same prompts, same scorers.

Category Bonsai 2 Qwen3.8 27B (FP16) Delta
Coding 81.58 82.17 -0.59
Math 96.57 97.06 -0.49
Vision 78.59 81.64 -3.05
Instruction Following 82.66 81.25 +1.41
Knowledge / Reasoning 83.95 86.66 -2.71
Agentic 77.57 79.74 -2.17
Aggregate 83.90 85.40 -1.50 (98.2% retention)

The aggregate 1.5-point drop on a 20-benchmark suite is the headline: 98.2% retention at one-ninth the footprint. Two rows tell the structural story. Vision drops 3.05 - ternary quantization hits the visual encoder harder than the language stack. Instruction Following gains 1.41 - quantization noise apparently regularizes the post-training toward the format-following objective. The two deltas cancel imperfectly in the aggregate, but they are not random. The retention is structural, not accidental.

Independent confirmation via BenchLM lands at 50.78/100, rank #113 of 490. Treat that as conservative - BenchLM’s coverage on Bonsai 2 is 22 of 446 benchmarks. Per-benchmark where it does cover: LiveCodeBench v6 90.1%, MATH-500 98.8%, GSM8K 96.7%, IFEval 91.3%, MMLU-Redux 89.1%, SWE-bench Verified 60.8%, BigCodeBench 58.1%, Terminal-Bench 2.1 52.8%, BFCL v3 74.9%, AA-LCR 77.0%. The numbers are not contradictory; they are two different sampling strategies on the same model.

The deployment envelope: 5.9GB vs 18GB vs 27GB

A standard 27B-class model at Q4_K_M sits around 18GB. At Q8_0 it sits around 27GB. At FP16 it is roughly 54GB - not deployable on a single consumer GPU. Bonsai 2 ships in two GGUF packings:

  • PTQ1_0: 5.95GB on disk, 1.75 bits/weight, packed ternary with FP16 group scales
  • PQ2_0: 7.21GB on disk, 2.13 bits/weight, ternary unpacked into 2-bit slots

Plus an MLX 2-bit build for Apple Silicon with custom ternary hybrid-attention kernels (separate repo, separate perf envelope). For the local-deployment conversation, the 5.95GB figure is the one that matters. On a 24GB consumer GPU - RTX 4090, RTX 5090, RX 7900 XTX - full offload leaves roughly 18GB for KV cache and context. On a 16GB card it leaves roughly 10GB. The model fits. The cache headroom is real.

Compared to the 2026-06-22 local-LLMs-2026 practical threshold analysis (which framed 7B Q4 as the practical consumer-GPU ceiling), Bonsai 2 just moved that ceiling by a factor of four. A 27B model running at full base-model quality on a card the previous ceiling assumed held a 7B. That is not a quantization tweak. That is a deployment category.

The efficiency table: 0.714 mWh/token on RTX 4090, 40% better than FP16 8B

Hardware Throughput Energy
RTX 5090 143 tok/s -
M5 Max (MLX 2-bit) 46.8 tok/s -
RTX 4090 - 0.714 mWh/token

The 0.714 mWh/token figure on the 4090 is the structural argument. Per PrismML’s own measurements, that is 40% more energy-efficient than an 8B model running at FP16. The argument is not “27B is faster than 27B at FP16” - it obviously is not. The argument is “27B at ternary uses less energy per token than 8B at FP16, because the bottleneck is memory bandwidth and ternary collapses the bytes the memory bus has to push.”

That is the right framing for the local-AI build-out conversation. The previous generation of quantization research optimized for “how few bits can we ship.” Bonsai 2’s framing is “how few joules per token can we run.” The shift from a storage problem to an energy problem is the structural change.

Training vs inference: the trick shrinks what you run, not what you build

This is the framing Startup Fortune published on the Bonsai 2 release and the one PrismML’s whitepaper converges on. Bonsai 2 was trained on Google v5 TPUs. Training compute is unchanged from the FP16 base. What collapsed is inference memory and inference energy. The model has 27.8B parameters. They are stored at 1.76 bits each with FP16 group-wise scaling. The runtime unpacks the ternary values on the fly during the matmul, never materializes the FP16 tensor in memory.

The structural read is deployment-unlock at a specific memory envelope, not market disruption. PrismML is not “killing NVIDIA.” NVIDIA’s datacenter training story is unchanged - TPU v5 still trains the base, and the ternary quant recipe is post-training quantization with a calibration pass, not a from-scratch retraining. NVIDIA’s consumer-GPU inference story is the one that just shifted: the same RTX 4090 that previously held a 7B at Q4 now holds a 27B at full base-model quality. That is a category expansion for the consumer-GPU installed base, not a category kill.

Meta’s Muse Glimmer 30B shipped day-0 in llama.cpp, vLLM, and ExecuTorch on August 11. That was the “30B multimodal agent on one consumer GPU” story. Bonsai 2 is the same structural argument with a sharper deployment envelope: same RTX 4090, 5.9GB instead of 18GB, full base-model quality instead of a 30B-from-7B distill. The two posts sit in the same lineage.

LM Studio walkthrough: GPU layers, system RAM, what 5.9GB actually requires

For anyone wanting to run Bonsai 2 locally, the deployment recipe is concrete. Download the GGUF from prism-ml/Ternary-Bonsai-2-27B-gguf on Hugging Face. Import into LM Studio (the PTQ1_0 5.95GB file is the one to start with; the PQ2_0 build is the higher-throughput variant at the cost of footprint). Start the local server on port 1234. The UI exposes the layer slider under “GPU Offload.”

For an 80-layer Qwen-class backbone, --gpu-layers 80 is full offload. On a 24GB card that leaves the 18GB KV-cache and context headroom discussed above. Partial offload is the realistic mid-spec option: --gpu-layers 40 pushes the first 40 layers to VRAM and lets the rest stream from system RAM. On a 16GB card with 32GB system RAM, that gives you 5.9GB in VRAM, ~10GB for KV cache, and the rest streamed at PCIe bandwidth. Throughput drops, but the model runs.

The system RAM requirement is the one that trips people up. Even with full GPU offload, the packed weights live in RAM until they are needed. VRAM holds the active layer weights and the K/V cache; RAM holds the rest of the packed weights and the unpacking buffers. For Bonsai 2 at 5.9GB on a 24GB GPU: full offload leaves ~18GB for KV cache and context, but a 262K context with 40+ concurrent requests can still need 16-24GB of system RAM as scratch. The rule of thumb is “1.5x to 4x the model footprint in system RAM, depending on concurrency.” Apple MLX handles this differently - on Apple Silicon the unified memory architecture means the same physical RAM serves both roles, which is why the M5 Max 46.8 tok/s figure is on the same memory budget.

The Bonsai 2 demo repo on GitHub (PrismML-Eng/Bonsai-demo) walks through a Cline coding-agent deployment on an RTX 5090 with full offload. That is the canonical reference if you want a known-good config to start from.

The retention curve: 95% in March, 98.2% in September

The lineage matters. Microsoft’s March 15, 2026 BitNet coverage framed BitNet as a research direction with a “100B-on-a-CPU” pitch that did not survive contact with deployment reality. Bonsai 1 in March landed at roughly 95% retention at the same 1.7-bit class. Bonsai 2 in September lands at 98.2% - 3 percentage points up in 6 months, using a different base model (Qwen3.8 instead of Qwen3.6) and a ternary-tuned training recipe. The retention curve is moving in the right direction at roughly half a percentage point per month.

What that implies for the next release at this cadence: Q1 2027 should land at 99% retention at the same envelope. At that point the gap to FP16 is noise-floor for most workloads. The structural argument - “why ship a 27B at 5.9GB instead of a 7B at 4GB” - flips on its head. The answer becomes “because the marginal hardware cost of running 27B at ternary is roughly the same as running 7B at Q4, and the capability gap between 7B and 27B at the same retention is still a capability gap.”

The ArXiv companion: 1.58-bit floor and the BITCOS 1.48 frontier

The companion paper “Breaking the 1.58-bit Barrier for Ternary LLMs” (ArXiv 2609.16338) frames the theoretical floor. Log2(3) = 1.585 bits per weight is the information-theoretic limit for ternary {-1, 0, +1}. Bonsai 2 lands at 1.76 - above the floor because the FP16 group scales carry per-group magnitude information that the ternary encoding itself cannot represent. The paper points at BITCOS packing as the next frontier: 1.48-bit via dictionary-coded scaling that compresses the group scales themselves. That is below the theoretical floor on a naive analysis because the packing exploits the redundancy in the scale distribution.

The 1.58-to-1.48 gap is where the next 12 months of ternary research will land. A 27B model at 1.48 bits/weight with a 7-bit scale dictionary is roughly 5.2GB on disk - functionally the same envelope as Bonsai 2, with a different representation. The structural story does not change at that point; the bytes just keep getting denser.

Sources