

this is interesting, it makes sense but I’ve never tested it


this is interesting, it makes sense but I’ve never tested it
That blog post doesn’t seem to talk about what changed in this release since v0.9.0 https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0
llama.cpp v1.0.0 removes the deprecated public API and promotes the NextN/speculative API to llama.h, adds OCP FP8 KV caches, a host-memory MoE expert cache with a configurable CPU share, and loading of external PLE shards. It adds three text architectures (Kimi-K3.1-Pro, Ripple and xing4_0), drama-3 TTS, and MageVL-Cowork and Kimi-K3.1 vision, hardens llama-server (SSRF, CORS, DNS rebinding, MCP tool cancellation) and adds a /v1/audio/speech endpoint, ships a redesigned web UI with chat groups and a markdown editor, and updates ggml to v1.0.0.
Some highlights:
I hope they start doing a separate bleeding edge branch and stable branch. Like how Linux has rolling distros and LTS distros.
it’s so close, I’m gonna Qwum!


I find the tuner script is a bit overly ambitious for VRAM usage, so I run it once to get the initial recommendation, and then I run it again with a --max-mib a few hundred MB lower than that.
Has anyone tried testing with non-recommended share values? I suspect it might be overly ambitious there too, and maybe adding +0.02 or even +0.05 might be a little faster
I guess I’ll be keeping an eye on this https://github.com/ggml-org/llama.cpp/commits/master/scripts/moe-cache-tune.py


It seems like it keeps getting better https://github.com/ggml-org/llama.cpp/pull/31888
can’t wait to revisit this in the next stable release, and hopefully with Qwen 4 if they release a small MoE like the 35b


Sure if compaction was free then I’d do it all the time. One advancement I’d like to see is MoE models that use fewer active parameters/experts when compacting. We already have DeepSeek 4.1 that uses a different number of active parameters depending on what its doing (8b for prefill and 16b for decode) so it doesn’t seem unreasonable to use fewer for compaction. It’d be like using a REAP model, but without having to load the model separately. Or even just use the MTP/draft model for compaction.


Qwen3-Coder?? What year is it?!
Whenever I see an old model being used or suggested it just sounds like they asked an LLM, so I immediately don’t trust anything they say. LLMs love suggesting old models.


7.72 TB, I don’t even have enough SSD storage for this in the entire house lol
if this is real then just the fact that it’s 14T is interesting info about the frontier models
makes me feel like China’s models are at least as efficient as the US frontier models, they’re only behind in hardware
EDIT: this seems to be fake https://huggingface.co/burner22/numbat/discussions/13


this is not a regression just a performance inefficiency


Which model GPU and how much VRAM? 16GB RAM limits your options a bit but maybe LFM 2.5 24B-A2B
If not, Gemma 4 e4b or Ling 3.0 Tiny are about as small and fast as you can get while still being decently smart.


I have 8GB VRAM and 32GB RAM, mostly using llama.cpp to run Tiel 35b at Q4, it’s ok at writing code in an agent as long as it can compile and run tests to iteratively fix its mistakes.
Though there’s been new optimized engines, like Strata, that could maybe allow you to run Qwen Flash Next at decent speeds.
There’s also Qwen 4 coming out soon that may give some good new options for us VRAM poors. Just stick to MoE models.


Usually people don’t use it themselves, you just go on huggingface and search for “Qwen 3.6 35b heretic” or something like that for a premade heretic model.
What are the specs on the computer?
This can happen because of tokenization. Tell it to execute code to do the letter counting and both models will succeed every time


Lol yeah someone tell them about local LLMs, open weights, and abliteration


ANTHROPIC IS CRACKING DOWN ON VIDEO GAME REWRITES 🚨
Claude Code picked up a new rule around Oct 27, sent from Anthropic’s servers under a flag called “tengu_heron_brook”. There’s no setting to turn it off.
The full rule: “Do build original games, games in a genre, and games from the user’s own designs and assets. Don’t build a recreation of a specific existing game, even if it is renamed or called original.”
Is this the end of game rewrites, modding and everything in between?



But it isn’t illegal, so why?


so typical of the American models to be the best at maximizing prices lol


I thought Kimi K3 was funny https://foodtruckbench.com/blog/kimi-k3
It almost seems like it was trying to be as helpful as possible by feeding as many people as it could, too nice for its own good


I agree, I was just pointing out that the issue you quoted earlier is easily avoided
I’m starting to think it’s not worth pushing past a share of 0.95, even 0.95 can have issues. It can take too long to shuffle out experts that aren’t being used anymore. I like to look at the predicted table and lean away from high shares. Like in my VRAM limited example:
predicted t/s, warm cache: cache MiB 0 0.5 0.8 0.9 0.95 0.98 896 23.2 31.6 39.4 42.8 44.7 45.7 1856 31.4 38.3 45.7 48.8 50.3 50.9 2496 36.9 42.5 49.1 51.8 53.0 53.4A share of 0.9 is more than fast enough, even 0.8 might be good. I was trying 0.98 for a while and it would often get stuck in a rut where it was doing like 10 t/s for a long time as it was only moving 2% of experts per turn.
Now all I need is llama.cpp to support an option similar to Strata’s
STRATA_ROUTE_PRIOR(like a runtime version of REAP models, it biases towards using cached experts by however much you want)And KV cache streaming https://github.com/ggml-org/llama.cpp/discussions/28216