#localllama — Public Fediverse posts

I'm going to idle in a Matrix room from now on for those who want to discuss local AI (LLMs as well as diffusion models and adjacents). I've noticed this is an area that can be quite difficult to get into and since I've spent a considerable amount of time on it myself I'll happily share ideas and practices with others.

https://matrix.to/#/#LocalLLaMa:argot.se

Local models allow using them for many different workflows without having to think of token costs - as well as knowing that sensitive data don't leave your premises.

#LLM #AI #LocalLLaMa

I'm going to idle in a Matrix room from now on for those who want to discuss local AI (LLMs as well as diffusion models and adjacents). I've noticed this is an area that can be quite difficult to get into and since I've spent a considerable amount of time on it myself I'll happily share ideas and practices with others.

https://matrix.to/#/#LocalLLaMa:argot.se

Local models allow using them for many different workflows without having to think of token costs - as well as knowing that sensitive data don't leave your premises.

#LLM #AI #LocalLLaMa

RT @SlimTradeyBaby: Google hat Gemma 4 auf 31 Milliarden Parameter begrenzt. Ein Entwickler sagte: „Scheiß darauf." Er duplizierte nicht einfach nur Schichten. Er führte eine Art neuronale Chirurgie durch: Er fügte völlig neue Schichten ein, die so konstruiert waren, dass sie zunächst wie perfekte Geisterkopien wirkten und die Ausgabe des Modells um genau null veränderten. Dies schuf leeren mentalen Raum innerhalb der KI, damit sie völlig neues Wissen (koreanisches Recht und fortgeschrittene MINT-Fächer) absorbieren konnte, ohne das bereits Gelernte zu zerstören. Dann trainierte er es. Die neuen Schichten blieben nicht passiv. Sie „erwachten" und trugen mehr bei als die ursprünglichen Schichten. Während Google auf Sicherheit setzt, baut das Open-Source-Underground im Stillen die Modelle, die Google nicht bauen wird. Das verändert alles. teddit.net/r/LocalLLaMA/comm… Link Von der LocalLLaMA-Community auf Reddit: Ich habe Gemma4-31B auf 44 Milliarden Parameter (88 Schichten) erweitert — da Google... Entdecke diesen Beitrag und mehr aus der LocalLLaMA-Community auf reddit.com

mehr auf Arint.info

#AICommunity #Gemma4 #LocalLLaMA #MachineLearning #NeuralSurgery #OpenSourceAI #arint_info

https://x.com/SlimTradeyBaby/status/2073385084380426608#m

EVO-X2 users? Is it worth it?

#localllama

EVO-X2 users? Is it worth it?

#localllama

So with this and smart use of LLM hosting (vLLM / Ollama etc) I can run the likes of lucidRAG (which needs GPUs) for FREE (well energy prices)...
Local LLM FTW!
#localllama

Google just released "QAT" versions of their Gemma 4 models. QAT stands for "yeah we know you people don't have enough VRAM so we trained the model knowing you'd quantize it down to 4 bits anyway" and apparently that makes a 4-bit QAT-model perform similar to an 8-bit quantized with previous methods.

This is a game-changer for running LLMs locally. As a first try I'm running unsloth's version of the 12b model released yesterday, and _without_ quantizing the KV-cache and with >128000 byte context it's not even filling up my 16GB VRAM. Prompt processing > 2000t/s and inference at >40 t/s.

https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF

#LLM #LocalLLaMa #Gemma4

Unsloth Qwen3.6: New coding model of interest if you run AI coding agents locally
https://unsloth.ai/docs/models/qwen3.6
#localllama #claudecode #coding #agents #qwen #ai #+

@LocalLLaMA Trùng hợp 2 CPU Xeon 2690 V3 + RAM 2133MHz, nâng tối đa 256GB để tự lưu trữ mô hình AI cỡ lớn (LLM). Có thể chạy mô hình 235B như Qwen3? Mời chia sẻ kinh nghiệm & đánh giá hiệu năng! #SelfHosting #LLM #AI #TựNhânViễn #Qwen3 #MáyChủAI #ThửTháchCôngNghệ #LocalLLaMA

https://www.reddit.com/r/LocalLLaMA/comments/1py4nuu/self_hosting_llm_on_multi_cpu_sys_ram_combo/

Xiaomi tung phiên bản mới **MiMo-V2-Flash (309B)**, модель với khả năng xử lý ngôn ngữ cực mạnh, tiến gần hơn đến công nghệ AI hàng đầu. Đây là bước tiến đáng chú ý trong lĩnh vực trí tuệ nhân tạo.

#Xiaomi #AI #TríTuệNhânTạo #CôngNghệ #MIMOV2Flash #TechNews #LocalLLaMA

https://www.reddit.com/r/LocalLLaMA/comments/1prjzoh/xiaomis_mimov2flash_309b_model_jumping_straight/

Exo 1.0 cho phép kết nối Macbook qua cáp Thunderbolt để chạy mô hình AI lớn. Không có giới hạn phần cứng, hỗ trợ đa thiết bị (M1, M2) khác nhau. Cổng Thunderbolt an toàn, tự động phát hiện thiết bị. #Exo10 #MLX #Macbook #AI #KếtNốiMáyTính #TechViet #LocalLLaMA #AppleSilicon #M1M2 #KỹThuậtViễnThông #HợpTácMáyTính

https://www.reddit.com/r/LocalLLaMA/comments/1pqfs1l/exo_10_means_you_can_cluster_mac_studios_for/

Ever wonder how tools analyse gigabytes of data, while you’re fighting to squeeze a few KB of code into an Ollama context window?

The trick isn’t bigger models.
It’s not giving the data to the LLM at all.

LLM → SQL → DuckDB → CSV-on-disk
Schema + samples only. Zero data leakage. Sub-100ms queries on 500MB+ files.

Local. Private. Scales past RAM.

Full C# walkthrough:
https://www.mostlylucid.net/blog/analysing-large-csv-files-with-local-llms

#llm #csharp #duckdb #ollama #localllama

Playing with compar:IA, the new LLM chatbot comparison arena created by the French government: https://comparia.beta.gouv.fr

I particularly like the "frugal" mode that allows you to compare two randomly chosen small/cheap models against each other! It's great for testing many different models suitable for local hosting on a particular task.

After each chat session, it also shows you an estimate of the energy consumed by each model!

#AI #LLM #FrugalComputing #LocalLlama #chatbot

Do you have any coder LLM recommendations that can run locally with 16 GBs of RAM, other than Qwen-2.5-Coder:7b?

#llm #ollama #localllama

Which VSCode extension you'd prefer to try local LLMs or through Openrouter etc. ?

#continue #cline #roo #localllama #vscode