#llama_cpp — Public Fediverse posts

I got Deepseek V4 work in Framework Desktop, an AMD Strix Halo PC with 128G memory, inside Lemonade server. I briefly tested code snippet generation, and result was pretty good. Downsides are that it's rather slow and context is short. But coming from a mini-PC it's still *very* impressive. The little Framework PC keeps giving.

Speed was about 15 tps, but the speed stays constantly there - even when context gets longer. I remember running Kimi K2 a year ago at 3 tps!

I tested code generation by asking it to implement an Angular module for OAUTH login client. I refined it through few iterations to add e.g. hardening and configuration features. Code quality was very good. Finally i asked it to write it out as plan.md, restarted and asked to generate code from the plan. Regenerated code was nearly identical to original round.

The server had some instability after chat grew to about 35k long (total 15k tokens). Nothing crashes but client showed an error that stream ended prematurely. The server log showed it finished though. Overall this was good experience, with some concern about actual max content length.

Model was unsloth/DeepSeek-V4-Flash-0731-GGUF with UD-IQ2_M quant. The Lemonade server couldn't run it out-of-the-box, complaining about unknown "Deepseek" architecture.
- I upgraded llama.cpp to a nightly build:
lemonade config set llamacpp.rocm_bin=b10230
- Reloaded the llama.cpp backend from Lemonade UI.
- File/Add model.
- Set run parameters: --flash-attn on --reasoning on -np 1 --ctx-checkpoints 0
- Max context: 65k

RAM usage was at 90GB, so there is still room for another model in parallel, or better quant. No crashes, even after several hours. Next step is try Hermes Studio with the model.
#homelab #AI #deepseek #framework #lemonade #unsloth #llama_cpp #hermes_agent #amd #rocm

I got Deepseek V4 work in Framework Desktop, an AMD Strix Halo PC with 128G memory, inside Lemonade server. I briefly tested code snippet generation, and result was pretty good. Downsides are that it's rather slow and context is short. But coming from a mini-PC it's still *very* impressive. The little Framework PC keeps giving.

Speed was about 15 tps, but the speed stays constantly there - even when context gets longer. I remember running Kimi K2 a year ago at 3 tps!

I tested code generation by asking it to implement an Angular module for OAUTH login client. I refined it through few iterations to add e.g. hardening and configuration features. Code quality was very good. Finally i asked it to write it out as plan.md, restarted and asked to generate code from the plan. Regenerated code was nearly identical to original round.

The server had some instability after chat grew to about 35k long (total 15k tokens). Nothing crashes but client showed an error that stream ended prematurely. The server log showed it finished though. Overall this was good experience, with some concern about actual max content length.

Model was unsloth/DeepSeek-V4-Flash-0731-GGUF with UD-IQ2_M quant. The Lemonade server couldn't run it out-of-the-box, complaining about unknown "Deepseek" architecture.
- I upgraded llama.cpp to a nightly build:
lemonade config set llamacpp.rocm_bin=b10230
- Reloaded the llama.cpp backend from Lemonade UI.
- File/Add model.
- Set run parameters: --flash-attn on --reasoning on -np 1 --ctx-checkpoints 0
- Max context: 65k

RAM usage was at 90GB, so there is still room for another model in parallel, or better quant. No crashes, even after several hours. Next step is try Hermes Studio with the model.
#homelab #AI #deepseek #framework #lemonade #unsloth #llama_cpp #hermes_agent #amd #rocm

小型ローカルLLM(9B)のコーディング精度は「ハーネスとモデルの相性」で決まる ― Claude Code系とhermes-agentを比較
https://qiita.com/nogataka/items/136bf8e446b13427d0e6?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #LLM #llama_cpp #ollama #AIエージェント #ClaudeCode

LM Studio 0.4.0
https://lmstudio.ai/blog/0.4.0
#ycombinator #local_ai #local_llm #gpt_oss #on_device_ai #run_local_ai #LM_Studio #Llama #Gemma #Qwen #DeepSeek #llama_cpp #mlx

LM Studio 0.4.0
https://lmstudio.ai/blog/0.4.0
#ycombinator #local_ai #local_llm #gpt_oss #on_device_ai #run_local_ai #LM_Studio #Llama #Gemma #Qwen #DeepSeek #llama_cpp #mlx

「vLLM vs llama.cpp」徹底比較：GPUサーバとローカルLLMの最適な選び方
https://qiita.com/softbase/items/033f11382117e46bca78?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #Python #CUDA #LLM #llama_cpp #vLLM

Best distro for #Framework destop 128G is #Bazzite. Add SSH service and a container running #llama_cpp and #Kimi Dev 72B. Now i can generate code, and there is still plenty of memory for long context.

And when getting bored, switch to play some #Steam games - on the same machine. 😊
#homelab #AI #gaming

中古GPU（RTX 3060/12GB）でローカルLLM検証-2 ～ llama.cpp で TinyLlama 1.1B を試す
https://qiita.com/nabe2030/items/15e7b6cffd46fafb34d4?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #CUDA #NVIDIA #RTX3060 #LLM #llama_cpp

#MistralSmall24B-Instruct is a really nice model to run locally for Coding Advice, Summarizing or Creative Writing.

With a recent #llama_cpp on a #GeForce #RTX4090 at Q8, the 24GB VRAM is tightly maxed out and I am seeing text generation at 7-9 token/s.

https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501

#MistralSmall24B-Instruct is a really nice model to run locally for Coding Advice, Summarizing or Creative Writing.

With a recent #llama_cpp on a #GeForce #RTX4090 at Q8, the 24GB VRAM is tightly maxed out and I am seeing text generation at 7-9 token/s.

https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501

Llama.cpp guide – Running LLMs locally on any hardware, from scratch
https://steelph0enix.github.io/posts/llama-cpp-guide/
#ycombinator #llama_cpp #llama #cpp #llm #building #running #guide #inference #local #scratch #hardware

Llama.cpp guide – Running LLMs locally on any hardware, from scratch
https://steelph0enix.github.io/posts/llama-cpp-guide/
#ycombinator #llama_cpp #llama #cpp #llm #building #running #guide #inference #local #scratch #hardware

Llama-3-ELYZA-JP-8Bを使ってStreamlitでサクッとLLMアプリ開発
https://qiita.com/sat01m0/items/9a7c07e80afa4b121e35?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #Streamlit #LLM #llama_cpp #ELYZA #Llama_3_ELYZA_JP_8B

Raspberry Pi 5 で高性能な日本語 LLM RakutenAI-7B-chat を動かす
https://qiita.com/susumuota/items/66585baa0ec8e38e3367?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #RaspberryPi #機械学習 #LLM #llama_cpp #raspberrypi5

Anyone happen to know the correct prompt format for #mixtral 8x7b? I'm not having luck with the [INST] style in llama.cpp. #llm #llama_cpp #MistralAI #mistral

Anyone happen to know the correct prompt format for #mixtral 8x7b? I'm not having luck with the [INST] style in llama.cpp. #llm #llama_cpp #MistralAI #mistral