home.social

#llamaserver — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #llamaserver, aggregated by home.social.

  1. I've been testing the new llama-server 1.0.0 release: the MoE optimizations really work well for my dual RTX 3060 machine: with Gemma-4-26b-a4b I get up to 90 tokens/s for small context sizes, a 20% speed-up.

    But that's not all: MTP works very well, and I get up to 130 tokens/s (small context) and still > 90 token/s for larger ones.

Share on Mastodon

Enter the server where you have an account.