The decision-model thread stopped being research today: llama.cpp merged it.
Top Signal
llama.cpp adds first-class decision model support
research to practice
r/LocalLLaMA
llama.cpp now supports decision models natively — small, cheap-to-decode models whose job is to emit a choice (route, classify, approve, pick a tool) rather than prose. Until today this thread lived in benchmarks (JevBench), bespoke distillations (JevStiller), and one-off sub-5B fine-tunes like jeff-qwen3.5-0.8b. Folding it into llama.cpp turns a research pattern into infrastructure you can serve from the runtime you already run. Why it matters: the expensive part of most agent loops is not generation, it's the dozens of small judgments — is this chunk relevant, which tool, retry or fail — currently paid for at frontier latency and frontier price. What to do: list the three highest-volume classification calls in your pipeline, benchmark a sub-2B decision model against your current model on your own labeled examples, and reserve the frontier model for the calls that genuinely need free-form output.
Read more →
Fast Signals
Redis creator ships ds4, a local LLM runner
new tool
HN Front Page
The creator of Redis released ds4 (dwarfstar.sh), a tool for running LLMs locally. Worth a look purely on authorship — this is a systems author with a long record of small, fast, dependency-light C, in a space currently dominated by sprawling forks. Install it and benchmark against your llama.cpp setup before committing to another runtime.
Link →
iPhone as a second GPU: 29-44% faster prefill on a MacBook
workflow
r/LocalLLaMA
A developer wired an iPhone into a 24GB MacBook as a compute and KV-cache peer, getting 29-44% faster prefill on Qwen3.8 27B and offloading part of the context window to the phone. The real signal is heterogeneous device pooling over a local link — treating idle Apple silicon as addressable memory. Bookmark for when VRAM, not FLOPs, is your ceiling.
Link →
192GB of non-CUDA inference from two Huawei Atlas cards
workflow
r/LocalLLaMA
A detailed writeup of building a local inference box on two Huawei Atlas 300I Duo cards — 96GB each, passive, cheap, and emphatically not drop-in CUDA — running Qwen3.8-flash-next with vLLM, including the porting work and benchmarks. This is the most concrete escape hatch from NVIDIA pricing posted this week. Read it before your next capex decision, with eyes open about the driver tax.
Link →
Strata hits 150-200 tok/s on a power-limited 5090 plus DDR5
workflow
r/LocalLLaMA
Qwen3.8-Flash-Next at IQ3_S with a 128k 8-bit context, decoding 150-200 tok/s with 5-6k prefill on one power-limited 5090 and 96GB of DDR5-6400. A reproducible configuration showing a large MoE is serviceable on a single consumer card when RAM offload is tuned. Copy the exact quant and context settings as a starting baseline.
Link →
ChatGPT can now host sites
platform change
HN Front Page
OpenAI shipped Sites in ChatGPT, putting page hosting inside the assistant itself. For builders this is a distribution surface question, not a tooling one: generated artifacts now get a URL inside the chat product where your users already are. Watch whether it gets an API before building anything on it.
Link →
llama.cpp PR fuses shared experts into MMVQ on CUDA
platform change
r/LocalLLaMA
A CUDA kernel change folds shared-expert matmuls into MMVQ, cutting per-token overhead for MoE models — which is now most of what local users run. Free decode speedup once merged, no config changes. Track the PR and rebuild rather than pinning an old release.
Link →
Qwen3.8-27B-Humanlike-Chat 2.0 adds tool calls
new tool
r/LocalLLaMA
A community fine-tune tuned to text like a person now handles tool calling and follows instructions better — previously the tradeoff was conversational tone or usable function calls, not both. Relevant if you ship anything with a persona that also has to do work. Grab the weights and test whether the tool-call reliability survives the voice.
Link →
Radar
Percepta Spotlight separates intelligence from memory
A claimed new architecture where knowledge and skills grow without touching model weights — continual learning as an external, writable store rather than a fine-tune. Unverified, but if the retrieval-free version of this works it changes how you think about model updates entirely.
Link →
ldraw-nova generates LEGO builds via LDraw codegen
An open-source tool that has models emit LDraw, the low-level LEGO assembly language, instead of images. The transferable idea is the one behind reladraw last week: give the model a deterministic, verifiable DSL as the output target and validation becomes free.
Link →
microsoft/FrogNano-4B-2609 appears on Hugging Face
A new 4B from Microsoft, posted without the usual launch cycle. Worth a quick eval slot if you run anything in the 4B class, where the practical ceiling keeps moving.
Link →
Peacebell: 11 months to a from-scratch domain SLM
One developer trained a small language model from scratch specialized on WWII, with a writeup of why not to just fine-tune. Useful data point on what solo pretraining actually costs in time if you are weighing a domain model.
Link →
Stratego falls to AI, reportedly on a small budget
A Nature paper beats the best human Stratego player — imperfect-information planning with most of the board hidden, done without a frontier-lab compute budget. The planning-under-hidden-state techniques are the part that eventually matters for agents.
Link →
Convergence Watch
decision models
TRENDING
1 mentions across r/LocalLLaMA
Fourth appearance in seven days, after home-trained sub-5B models beating frontier on JevBench, the jeff-qwen3.5-0.8b release, and HN coverage. Today it crosses from model releases into runtime support. The pattern is consolidating fast: small models own the judgment calls, frontier models own generation.
llama.cpp
TRENDING
3 mentions across r/LocalLLaMA
Two substantive changes land the same day — decision model support and fused shared experts on CUDA — on top of last week's GLM-5.3-Flash PR and MTP work for an unreleased Qwen. Features now arrive upstream before model vendors ship weights. Pin to a commit, not a release, and rebuild often.
qwen3.8
TRENDING
5 mentions across r/LocalLLaMA
Every local-inference experiment today uses it as the reference model: iPhone prefill offload, the Ascend build, the Strata config, a new fine-tune, and quant comparisons. It is now the control variable the community benchmarks against, which makes its quant behavior worth knowing in detail.
local hardware capex
TRENDING
4 mentions across r/LocalLLaMA
A 64GB DGX Spark alongside a $6,950 128GB, 5090 purchases now requiring no-export paperwork, Huawei Atlas cards as a CUDA escape hatch, and phone-as-GPU offload. Fourth day in a week. Supply and pricing pressure is pushing real engineering effort into heterogeneous and offload-heavy serving.