https://www.wacoca.com/news/2985798/ ローカルAI基盤「llama.cpp」が初のv1.0.0、3年半でAPI安定性を明示 – BigGo ファイナンス #CondaForge #Drama3 #ggml #KimiK31Pro #LlamaServer #LlamaCpp #MageVLCowork #Science&Technology #ScienceNews #TechnologyNews #Vulkan #WebGPU #WinGet #テクノロジー #科学 #科学&テクノロジー
#llamaserver — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #llamaserver, aggregated by home.social.
-
I've been testing the new llama-server 1.0.0 release: the MoE optimizations really work well for my dual RTX 3060 machine: with Gemma-4-26b-a4b I get up to 90 tokens/s for small context sizes, a 20% speed-up.
But that's not all: MTP works very well, and I get up to 130 tokens/s (small context) and still > 90 token/s for larger ones.