The memory wall cracked from four directions today, and none of the cracks came from a frontier lab.
Top Signal
llama.cpp PR caches hot MoE experts in VRAM: 33 to 56 tok/s on 8GB
research to practice
r/LocalLLaMA
llama.cpp PR #26563 adds a usage heatmap that tracks which MoE experts actually get selected during inference, then pins the frequently-used ones in VRAM while cold experts stay on CPU. Reported result: 33 to 56 tok/s on an 8GB card. This is a smarter answer to the offload problem than the current all-or-nothing `-ngl` and `--n-cpu-moe` flags, which force you to choose between fitting the model and running it fast. Because expert selection is skewed in practice, a small hot set buys most of the speedup. What to do: if you run any MoE locally (Ling-3.0-flash, DeepSeek V4 Flash, Qwen3.8, GLM), build the PR branch and re-benchmark before you buy more VRAM. Also re-run any offload tuning you did in the last month, since the optimal split changes completely under expert-level caching. Watch for merge, and expect quantization advice to shift with it.
Read more →
Fast Signals
Swiftlet claims an 80B Qwen in 4.3GB of RAM, 35B on an iPhone
new tool
HN Show
A Swift runtime that runs an 80B Qwen in 4.3GB on a Mac and a 35B on an iPhone, at 298 points on HN. Treat the numbers as unverified: this is the fourth extreme-compression project in a week (turbofieldfare twice, now this), and none have published quality deltas against the unquantized baseline. Worth cloning if on-device is on your roadmap, but benchmark output quality yourself before you plan around it.
Link →
Mistral ships Shieldstral, a 3B open-weights multimodal moderation model
new tool
HN Front Page, r/LocalLLaMA
A small open-weights classifier for moderating text and images, hitting both HN's front page and r/LocalLLaMA the same day. The builder angle is cost: a 3B guardrail you self-host replaces per-call moderation API spend and removes a network hop from your request path. If you currently route user content through a hosted moderation endpoint, this is a same-week swap to evaluate.
Link →
MiniMax-H3 gets an MLX port two days after release
new tool
Simon Willison
MiniMax's omni-modal H3 now runs on Apple Silicon via PipeNetwork/minimax-h3-mlx, closing the gap between a Chinese lab's open release and a local Mac dev loop to roughly 48 hours. That turnaround is itself the signal: open omni-modal weights are becoming testable on a laptop before the hosted APIs even list them. Pull it if you were waiting on audio or video input to prototype locally.
Link →
Shai-Hulud worm compromises Keyv and friends on npm
platform change
HN Front Page
Aikido reports an active supply chain attack that hit Keyv, a package buried deep in the transitive dependency tree of most Node tooling, including plenty of AI SDKs. Audit your lockfiles today and pin, since agent-generated code that adds dependencies without review widens this blast radius considerably. Concrete action: run your dependency scanner against the affected version list before your next deploy.
Link →
Yegge: Gas Town worked through Opus 4.6, then fell apart on 4.7
workflow
Simon Willison
Yegge reports his agent harness broke on a model upgrade after working brilliantly on the prior version, which is the failure mode nobody budgets for. Harnesses encode assumptions about how a specific model plans and recovers, so a better model can invalidate them. Pair this with last week's finding that harness design alone swung a 4B model from 60% to 82%: build a regression suite for your harness, not just your prompts.
Link →
Ling-3.0-flash weights land on HF under MIT with an official FP8
new tool
r/LocalLLaMA
inclusionAI published BF16 plus a first-party FP8 build, which matters more than the license: an official FP8 removes the guesswork of community quants with unknown calibration data. Yesterday's report that Ling-3.0-flash fixed bugs Qwen3.6-27B could not now has downloadable weights behind it. Grab the FP8 and run it against your own coding evals.
Link →
Soup fine-tunes an 8B model on a 4GB laptop GPU
new tool
HN Show
A Show HN project that pushes 8B fine-tuning onto hardware most developers already own, no rented A100 required. If you have been putting off a domain fine-tune because of GPU cost, this changes the experiment budget from dollars to an afternoon. Low star count, so read the training config before trusting the claim.
Link →
Radar
firecrawl/pdf-inspector classifies PDFs before you parse
A Rust library that detects scanned versus text-based PDFs so you can route to OCR only when needed. Directly complements yesterday's MinerU vs granite-docling vs PaddleOCR-VL bake-off: cheap classification in front of an expensive parser is the missing layer in most ingestion pipelines.
Link →
Mach-1 Additive claims 95% of Qwen 3.6 35B at 10x smaller
An r/LocalLLaMA thread asking whether anyone has actually tried it, which is the interesting part: nobody had. If the claim survives independent testing it resets small-model expectations, so it is worth a bookmark rather than a bet.
Link →
2.6B model does tool calling with 128K context at 30 tok/s on a phone
Tool calling plus long context on-device is the threshold where phone-local agents stop being a demo. If it holds up, the mobile agent story stops depending on a network round trip to a frontier model.
Link →
livekit/agents trending as realtime voice tooling consolidates
A framework for realtime voice agents climbing GitHub Trending, following NVIDIA's 11B full-duplex model and Hugging Face's local speech-to-speech pipeline last week. Voice is quietly assembling a full open stack; this is the orchestration piece.
Link →
Gemma 4 running in 500MB
Another entry in this week's compression race, with no published quality comparison. Watch the category rather than any single claim: the floor for useful local models is dropping faster than the evals can validate.
Link →
Convergence Watch
on-device model compression
TRENDING
6 mentions across r/LocalLLaMA, HN Show, Simon Willison
Six independent projects across three sources today attack the same wall: expert caching, Swiftlet, Gemma 4 in 500MB, Mach-1, Soup, and the MLX port. Combined with last week's turbofieldfare claims, this is now the dominant thread in open AI. The bottleneck has shifted from model quality to memory, and the fixes are coming from individuals, not labs. Verify claims independently, none publish quality deltas.
quantization
TRENDING
4 mentions across r/LocalLLaMA, HN Show
Third appearance in a week, but the framing changed. Monday's Qwen3.6-27B case study showed knowledge degrades nonlinearly under quantization; today's releases pair that caution with an official FP8 from inclusionAI. First-party quants are becoming the differentiator, since they carry known calibration data. Prefer official quants over community ones where both exist.
kimi k3
TRENDING
1 mentions across r/LocalLLaMA
Present five of the last seven days, but coverage has narrowed from capability debate to hardware logistics: today it is one post about a 16x GB10 cluster hitting 20+ tok/s. That is a maturing entity, not a rising one. The useful takeaway is the cost floor for self-hosting K3 at usable speed, not the model itself.