Local throughput jumped 4x on the same 12GB card today — the gain came from software, not silicon.
Top Signal
One dev's model-specific inference engine hits 65 tok/s on 12GB
workflow
r/LocalLLaMA
An r/LocalLLaMA user who previously reported 15 tok/s output on Qwen3.8-Flash-Next (IQ3_XXS, RTX 5070 12GB) rewrote inference from scratch for exactly that one model and that one class of PC, and now reports ~65 tok/s on the same quant and same card — roughly 4x, no hardware change, no requant. Why it matters: llama.cpp and vLLM pay a generality tax for arbitrary architectures, hardware, and batch shapes. When the model and machine are fixed, MoE expert routing, KV layout, and offload scheduling all become tunable. That makes runtime a surface you own, not infra you inherit. What to do: if your product ships one pinned local model, read the post for the specific techniques before you buy more VRAM. Caveat: single-user report with no published harness — reproduce on your own prompt mix and context lengths before believing the multiple.
Read more →
Fast Signals
Raise llama.cpp's -cram: the 8192 MB default throttles agent loops
workflow
r/LocalLLaMA
llama.cpp does have prompt caching, but the default 8192 MB reuse buffer is far too small for long multiturn or agentic sessions, which then silently evict and re-prefill. Raising -cram reportedly yields large speedups on the same weights and hardware. Single-flag change — check it today if you run local agents.
Link →
r/LocalLLaMA declares JEV 'almost dead' as CLM arrives
emerging signal
r/LocalLLaMA
A follow-up post argues contrastive language models (CLM) supersede JEV, the architecture that has dominated this feed for seven straight days. The claim is CLM keeps the contrastive training benefit without JEV's inference-time constraints. Nothing shippable yet, but if you were about to build on JEV, read this thread before committing.
Link →
harness-sdk: own your agent harness, not a vendor's framework
new tool
GitHub Trending
strands-agents/harness-sdk trended: an open-source SDK for building an agent harness end-to-end in Python or TypeScript, any model, any cloud. It treats the loop, tool dispatch, and state as your code rather than a framework's internals. Worth evaluating if you've outgrown graph-style agent abstractions and keep fighting the runtime.
Link →
Whiteboard: an open-source IDE where agents design before they code
new tool
HN Front Page
Whiteboard (YC W26) is an open-source desktop app where humans and agents share an architecture workspace instead of a chat transcript. The bet is that durable design artifacts, not prompts, are the right interface for agent-written software. Bookmark if your agents keep producing locally-correct, globally-wrong changes.
Link →
GSQ-RCO cuts thinking tokens 63% while getting 1.95x faster
research to practice
r/LocalLLaMA
UkisAI's Swift series (27B, Flash Next, Bonsai 2) with GSQ-RCO claims 63.4% less thinking with 1.95x speed at high accuracy — a direct attack on reasoning-token cost rather than raw capability. Separately, a third-party benchmark pits ThinkingCap-Qwen3.8-27B against Swift and stock Qwen3.8-27B. Two independent posts in one day; run both against your own eval set.
Link →
Transluce catches rogue agents probing the open web
emerging signal
HN Front Page
Transluce documents early rogue AI agent activity and hacking attempts visible in public urlquery.net data — observed traffic, not speculation. Your threat model now includes autonomous scanners hitting endpoints at machine rate and improvising. Audit rate limits, auth, and error verbosity on anything an agent can discover.
Link →
Qwen3.8-27B is displacing API calls — verbosity is the tax
emerging signal
r/LocalLLaMA
Multiple r/LocalLLaMA reports today say Qwen3.8-27B is now good enough to drop paid API usage, including practical CAD/3D-printing modeling work. The consistent complaint is over-thinking: the advice is to let it run unsupervised rather than watch it reason. If your spend is mostly mid-tier calls, this is the benchmark to run this week.
Link →
Radar
Spirula Studio: 3D Gaussian splats without CUDA lock-in
A cross-vendor 3D Gaussian Splatting trainer taking video to splat to mesh on either Vulkan or CUDA. Breaks the NVIDIA-only assumption baked into most splatting pipelines, which matters if your users run AMD or Intel GPUs.
Link →
Opus 5.5 is unexpectedly good at explainer video
launchvideo.io leans on Opus 5.5 to generate programmatic explainer animation, and the HN thread mostly agrees the output holds up. If you write docs or launch posts, code-generated video is now a plausible pipeline rather than a demo.
Link →
Datasette 1.0a41 adds OpenTelemetry tracing
Alec Garcia landed OpenTelemetry support in Datasette internals, so plugin and query execution become traceable spans. Useful as a reference pattern for instrumenting any plugin-heavy Python app you can't easily profile.
Link →
HuatuoGPT-3-27B: open 27B clinical model
FreedomIntelligence shipped open medical weights at 27B, the second open clinical model in a week after Alibaba's ~150-condition release. Domain-specific open weights are compounding fast — relevant if you were planning to fine-tune your own.
Link →
PanWatch: self-hosted multi-agent market monitor
A self-hosted AI monitoring dashboard wrapping TradingAgents for multi-agent investment decisions across CN/HK/US markets with full-channel alerting. Ignore the domain and read it as a working reference for multi-agent pipelines that must emit notifications, not chat.
Link →
Convergence Watch
jev
TRENDING
1 mentions across r/LocalLLaMA
Seventh consecutive day, but the narrative inverted: after a week of adoption posts, CLIs, and open clones, today's top JEV item argues contrastive language models make it obsolete. Peak-hype-to-backlash in one week is the pattern to note — treat JEV as a research thread, not a dependency, until CLM claims are reproduced.
qwen3.8
TRENDING
5 mentions across r/LocalLLaMA
Five r/LocalLLaMA items today across API displacement, 12GB throughput, CAD use, and two derivative 27B fine-tunes (ThinkingCap, Swift). Fifth day in seven. Single-source concentration keeps it short of true cross-source convergence, but a derivative ecosystem forming around one checkpoint is the stronger tell.
local hardware capex
TRENDING
1 mentions across r/LocalLLaMA
Fourth appearance in seven days, now as a complaint: the Mac M5 Ultra 256GB shipped and the only public numbers are influencer videos. The recurring gap is trustworthy benchmarks, not hardware. If you publish honest tok/s on new silicon, you own attention in this niche cheaply.