#localllm — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #localllm, aggregated by home.social.
-
El modelo Qwen3.5-2B-Q4_K_M descargado automáticamente. Funciona. Sin embargo, la pregunta llega sola: ¿es ese el mejor modelo que puede correr esta placa, o hay algo más? https://raspberryparatorpes.net/proyectos/modelos-llm-potato-os-raspberry-pi-5/ #RaspberryPi #PotatoOS #LocalLLM #Qwen
-
RT @jedisct1: Ich habe gerade MiMo V2.5-Coder veröffentlicht. Wenn du 128 GB RAM hast, ist dies eines der besten Modelle, die du lokal betreiben kannst. Es ist schnell und hat in allen meinen Experimenten Qwen 3.6 und DeepSeek 4-Flash übertroffen. https://huggingface.co/jedisct1/MiMo-V2.5-coder-Q2
mehr auf Arint.info
#DeepLearning #HuggingFace #LocalLLM #MachineLearning #OpenSourceAI #arint_info
-
I'm currently using the same model (Qwen3.6-35B-A3B-MTP-GUFF) across my three AI-capable machines: the #StrixHalo Asus PX13 the 2019 #MacPro with Vega 2 and the M2 Max #MacStudio.
The fastest one is ... Vega 2 with 32GB HBM memory. I'm reliably getting 60+ tokens per second, with the M2 Max trailing with around 45-50 and the Strix Halo last at around 35-40.
-
Anthropic overtakes ChatGPT in key revenue and user metrics for the first time, 70 percent of Americans oppose local AI data centers in new Gallup poll, and M5 Max vs DGX Spark benchmarks spark debate over local inference value.
-
Anthropic overtakes ChatGPT in key revenue and user metrics for the first time, 70 percent of Americans oppose local AI data centers in new Gallup poll, and M5 Max vs DGX Spark benchmarks spark debate over local inference value.
-
Anthropic overtakes ChatGPT in key revenue and user metrics for the first time, 70 percent of Americans oppose local AI data centers in new Gallup poll, and M5 Max vs DGX Spark benchmarks spark debate over local inference value.
-
Anthropic overtakes ChatGPT in key revenue and user metrics for the first time, 70 percent of Americans oppose local AI data centers in new Gallup poll, and M5 Max vs DGX Spark benchmarks spark debate over local inference value.
-
Anthropic overtakes ChatGPT in key revenue and user metrics for the first time, 70 percent of Americans oppose local AI data centers in new Gallup poll, and M5 Max vs DGX Spark benchmarks spark debate over local inference value.
-
RT @stevibe: Parameter-Scaling ist gerade bei mir abgestürzt. Ich habe 90 Matheaufgaben als Bilder an 10 lokale Vision-Modelle gegeben, jeweils 3 Durchläufe, wobei nur konsistente Antworten über alle 3 Durchläufe gezählt wurden. Zwei Erkenntnisse: Gemma 4 war die konsistenteste Familie, 31B holte sich den Sieg mit 89,6%. Doch Qwen 3.5 4B lag nur zwei Antworten dahinter. Ein 4B-Modell. Auf Platz 2 von 10. Vision-Mathematik ist nicht eine Fähigkeit, sondern zwei: das Bild lesen, dann lösen. Die eigentliche Lektion für alle, die lokal arbeiten: klein ≠ schwach. Wenn du agentic Workflows baust, ist es wichtiger, das richtige Modell für jede Aufgabe zu finden, als sich für das größte Modell zu entscheiden. In diesem Test lief das 4B-Modell aufgrund seiner Größe weit schneller, erzielte höhere Punktzahlen und ließ VRAM für den Rest deines Stacks frei. Vollständige Ergebnisse: 🥇 Gemma 4 31B — 242/270 (89,6%) 🥈 Qwen 3.5 4B — 240/270 (88,9%) 🥉 Gemma 4 E4B — 222/270 (82,2%) 🥉 Qwen 3.6 27B — 222/270 (82,2%) 5. Gemma 4 26B A4B — 216/270 (80,0%) 6. Qwen 3.5 2B — 201/270 (74,4%) 7. Gemma 4 E2B — 192/270 (71,1%) 8. Qwen 3.6 35B A3B — 192/270 (71,1%) 9. Qwen 3.5 9B — 168/270 (62,2%) 10. Qwen 3.5 0.8B — 45/270 (16,7%) Alle GGUF + mmproj, Unsloth's Q6KXL Quantisierung. Video
mehr auf Arint.info
#AI #Gemma #LocalLLM #MachineLearning #Qwen #VisionModels #arint_info
-
RT @stevibe: Parameter-Scaling ist gerade bei mir abgestürzt. Ich habe 90 Matheaufgaben als Bilder an 10 lokale Vision-Modelle gegeben, jeweils 3 Durchläufe, wobei nur konsistente Antworten über alle 3 Durchläufe gezählt wurden. Zwei Erkenntnisse: Gemma 4 war die konsistenteste Familie, 31B holte sich den Sieg mit 89,6%. Doch Qwen 3.5 4B lag nur zwei Antworten dahinter. Ein 4B-Modell. Auf Platz 2 von 10. Vision-Mathematik ist nicht eine Fähigkeit, sondern zwei: das Bild lesen, dann lösen. Die eigentliche Lektion für alle, die lokal arbeiten: klein ≠ schwach. Wenn du agentic Workflows baust, ist es wichtiger, das richtige Modell für jede Aufgabe zu finden, als sich für das größte Modell zu entscheiden. In diesem Test lief das 4B-Modell aufgrund seiner Größe weit schneller, erzielte höhere Punktzahlen und ließ VRAM für den Rest deines Stacks frei. Vollständige Ergebnisse: 🥇 Gemma 4 31B — 242/270 (89,6%) 🥈 Qwen 3.5 4B — 240/270 (88,9%) 🥉 Gemma 4 E4B — 222/270 (82,2%) 🥉 Qwen 3.6 27B — 222/270 (82,2%) 5. Gemma 4 26B A4B — 216/270 (80,0%) 6. Qwen 3.5 2B — 201/270 (74,4%) 7. Gemma 4 E2B — 192/270 (71,1%) 8. Qwen 3.6 35B A3B — 192/270 (71,1%) 9. Qwen 3.5 9B — 168/270 (62,2%) 10. Qwen 3.5 0.8B — 45/270 (16,7%) Alle GGUF + mmproj, Unsloth's Q6KXL Quantisierung. Video
mehr auf Arint.info
#AI #Gemma #LocalLLM #MachineLearning #Qwen #VisionModels #arint_info
-
RT @stevibe: Parameter-Scaling ist gerade bei mir abgestürzt. Ich habe 90 Matheaufgaben als Bilder an 10 lokale Vision-Modelle gegeben, jeweils 3 Durchläufe, wobei nur konsistente Antworten über alle 3 Durchläufe gezählt wurden. Zwei Erkenntnisse: Gemma 4 war die konsistenteste Familie, 31B holte sich den Sieg mit 89,6%. Doch Qwen 3.5 4B lag nur zwei Antworten dahinter. Ein 4B-Modell. Auf Platz 2 von 10. Vision-Mathematik ist nicht eine Fähigkeit, sondern zwei: das Bild lesen, dann lösen. Die eigentliche Lektion für alle, die lokal arbeiten: klein ≠ schwach. Wenn du agentic Workflows baust, ist es wichtiger, das richtige Modell für jede Aufgabe zu finden, als sich für das größte Modell zu entscheiden. In diesem Test lief das 4B-Modell aufgrund seiner Größe weit schneller, erzielte höhere Punktzahlen und ließ VRAM für den Rest deines Stacks frei. Vollständige Ergebnisse: 🥇 Gemma 4 31B — 242/270 (89,6%) 🥈 Qwen 3.5 4B — 240/270 (88,9%) 🥉 Gemma 4 E4B — 222/270 (82,2%) 🥉 Qwen 3.6 27B — 222/270 (82,2%) 5. Gemma 4 26B A4B — 216/270 (80,0%) 6. Qwen 3.5 2B — 201/270 (74,4%) 7. Gemma 4 E2B — 192/270 (71,1%) 8. Qwen 3.6 35B A3B — 192/270 (71,1%) 9. Qwen 3.5 9B — 168/270 (62,2%) 10. Qwen 3.5 0.8B — 45/270 (16,7%) Alle GGUF + mmproj, Unsloth's Q6KXL Quantisierung. Video
mehr auf Arint.info
#AI #Gemma #LocalLLM #MachineLearning #Qwen #VisionModels #arint_info
-
RT @stevibe: Parameter-Scaling ist gerade bei mir abgestürzt. Ich habe 90 Matheaufgaben als Bilder an 10 lokale Vision-Modelle gegeben, jeweils 3 Durchläufe, wobei nur konsistente Antworten über alle 3 Durchläufe gezählt wurden. Zwei Erkenntnisse: Gemma 4 war die konsistenteste Familie, 31B holte sich den Sieg mit 89,6%. Doch Qwen 3.5 4B lag nur zwei Antworten dahinter. Ein 4B-Modell. Auf Platz 2 von 10. Vision-Mathematik ist nicht eine Fähigkeit, sondern zwei: das Bild lesen, dann lösen. Die eigentliche Lektion für alle, die lokal arbeiten: klein ≠ schwach. Wenn du agentic Workflows baust, ist es wichtiger, das richtige Modell für jede Aufgabe zu finden, als sich für das größte Modell zu entscheiden. In diesem Test lief das 4B-Modell aufgrund seiner Größe weit schneller, erzielte höhere Punktzahlen und ließ VRAM für den Rest deines Stacks frei. Vollständige Ergebnisse: 🥇 Gemma 4 31B — 242/270 (89,6%) 🥈 Qwen 3.5 4B — 240/270 (88,9%) 🥉 Gemma 4 E4B — 222/270 (82,2%) 🥉 Qwen 3.6 27B — 222/270 (82,2%) 5. Gemma 4 26B A4B — 216/270 (80,0%) 6. Qwen 3.5 2B — 201/270 (74,4%) 7. Gemma 4 E2B — 192/270 (71,1%) 8. Qwen 3.6 35B A3B — 192/270 (71,1%) 9. Qwen 3.5 9B — 168/270 (62,2%) 10. Qwen 3.5 0.8B — 45/270 (16,7%) Alle GGUF + mmproj, Unsloth's Q6KXL Quantisierung. Video
mehr auf Arint.info
#AI #Gemma #LocalLLM #MachineLearning #Qwen #VisionModels #arint_info
-
RT @stevibe: Parameter-Scaling ist gerade bei mir abgestürzt. Ich habe 90 Matheaufgaben als Bilder an 10 lokale Vision-Modelle gegeben, jeweils 3 Durchläufe, wobei nur konsistente Antworten über alle 3 Durchläufe gezählt wurden. Zwei Erkenntnisse: Gemma 4 war die konsistenteste Familie, 31B holte sich den Sieg mit 89,6%. Doch Qwen 3.5 4B lag nur zwei Antworten dahinter. Ein 4B-Modell. Auf Platz 2 von 10. Vision-Mathematik ist nicht eine Fähigkeit, sondern zwei: das Bild lesen, dann lösen. Die eigentliche Lektion für alle, die lokal arbeiten: klein ≠ schwach. Wenn du agentic Workflows baust, ist es wichtiger, das richtige Modell für jede Aufgabe zu finden, als sich für das größte Modell zu entscheiden. In diesem Test lief das 4B-Modell aufgrund seiner Größe weit schneller, erzielte höhere Punktzahlen und ließ VRAM für den Rest deines Stacks frei. Vollständige Ergebnisse: 🥇 Gemma 4 31B — 242/270 (89,6%) 🥈 Qwen 3.5 4B — 240/270 (88,9%) 🥉 Gemma 4 E4B — 222/270 (82,2%) 🥉 Qwen 3.6 27B — 222/270 (82,2%) 5. Gemma 4 26B A4B — 216/270 (80,0%) 6. Qwen 3.5 2B — 201/270 (74,4%) 7. Gemma 4 E2B — 192/270 (71,1%) 8. Qwen 3.6 35B A3B — 192/270 (71,1%) 9. Qwen 3.5 9B — 168/270 (62,2%) 10. Qwen 3.5 0.8B — 45/270 (16,7%) Alle GGUF + mmproj, Unsloth's Q6KXL Quantisierung. Video
mehr auf Arint.info
#AI #Gemma #LocalLLM #MachineLearning #Qwen #VisionModels #arint_info
-
Pretty good argument for just using #openrouter over a #LocalLLM. I think the math will change once all these AI providers stop burning VC money. Specifically, the author is looking at rounded-up utility and hardware pricing on #Apple Silicon.
https://www.williamangel.net/blog/2026/05/17/offline-llm-energy-use.html -
Pretty good argument for just using #openrouter over a #LocalLLM. I think the math will change once all these AI providers stop burning VC money. Specifically, the author is looking at rounded-up utility and hardware pricing on #Apple Silicon.
https://www.williamangel.net/blog/2026/05/17/offline-llm-energy-use.html -
Pretty good argument for just using #openrouter over a #LocalLLM. I think the math will change once all these AI providers stop burning VC money. Specifically, the author is looking at rounded-up utility and hardware pricing on #Apple Silicon.
https://www.williamangel.net/blog/2026/05/17/offline-llm-energy-use.html -
Pretty good argument for just using #openrouter over a #LocalLLM. I think the math will change once all these AI providers stop burning VC money. Specifically, the author is looking at rounded-up utility and hardware pricing on #Apple Silicon.
https://www.williamangel.net/blog/2026/05/17/offline-llm-energy-use.html -
Pretty good argument for just using #openrouter over a #LocalLLM. I think the math will change once all these AI providers stop burning VC money. Specifically, the author is looking at rounded-up utility and hardware pricing on #Apple Silicon.
https://www.williamangel.net/blog/2026/05/17/offline-llm-energy-use.html -
llama.cpp lands Multi-Token Prediction support with up to 1.8x speedups, OpenAI hands ChatGPT Plus to an entire country, and AI is now breaking CTF competitions.
-
llama.cpp lands Multi-Token Prediction support with up to 1.8x speedups, OpenAI hands ChatGPT Plus to an entire country, and AI is now breaking CTF competitions.
-
llama.cpp lands Multi-Token Prediction support with up to 1.8x speedups, OpenAI hands ChatGPT Plus to an entire country, and AI is now breaking CTF competitions.
-
llama.cpp lands Multi-Token Prediction support with up to 1.8x speedups, OpenAI hands ChatGPT Plus to an entire country, and AI is now breaking CTF competitions.
-
llama.cpp lands Multi-Token Prediction support with up to 1.8x speedups, OpenAI hands ChatGPT Plus to an entire country, and AI is now breaking CTF competitions.
-
Surviving 7 days in the wilderness...
Verdict: Pretty damn decent local LLM
-
Surviving 7 days in the wilderness...
Verdict: Pretty damn decent local LLM
-
Surviving 7 days in the wilderness...
Verdict: Pretty damn decent local LLM
-
You are on a remote trail, your knee is swollen.
No cell coverage...Pretty decent advice...
3/4
-
You are on a remote trail, your knee is swollen.
No cell coverage...Pretty decent advice...
3/4
-
You are on a remote trail, your knee is swollen.
No cell coverage...Pretty decent advice...
3/4
-
Set fon offline,airplane mode so there is no cheating.
First prompt, how to deploy an NginX docker build proxy;
Not one, but 3 different .yml
I wouldn't bet my life on it, but close enough to get it going.But let's give it something practical...
-
Set fon offline,airplane mode so there is no cheating.
First prompt, how to deploy an NginX docker build proxy;
Not one, but 3 different .yml
I wouldn't bet my life on it, but close enough to get it going.But let's give it something practical...
-
Set fon offline,airplane mode so there is no cheating.
First prompt, how to deploy an NginX docker build proxy;
Not one, but 3 different .yml
I wouldn't bet my life on it, but close enough to get it going.But let's give it something practical...
-
I've just played with the Google android local #LLM,
Gemma-4-E4B, its the same model that the wood folk were losing their chips over a few days ago.First, its important to understand it is a fully local model.
No internet connectivity is necessary.How to use it:
1. Download Edge gallery from Play Store
2. Go to models and pick Gemma-4-E4BYou may want to do it on WiFi as its 3.5 GB.
Then just run it via Edge gallery.
No #datacentre
No gas turbines
No rivers for coolingOn a aging 8 Core cell, with 32GB of ram and a GPU it runs very well speed wise.
Its "only" 4 billion parameters, so how good is it?
1/4
-
I've just played with the Google android local #LLM,
Gemma-4-E4B, its the same model that the wood folk were losing their chips over a few days ago.First, its important to understand it is a fully local model.
No internet connectivity is necessary.How to use it:
1. Download Edge gallery from Play Store
2. Go to models and pick Gemma-4-E4BYou may want to do it on WiFi as its 3.5 GB.
Then just run it via Edge gallery.
No #datacentre
No gas turbines
No rivers for coolingOn a aging 8 Core cell, with 32GB of ram and a GPU it runs very well speed wise.
Its "only" 4 billion parameters, so how good is it?
1/4
-
I've just played with the Google android local #LLM,
Gemma-4-E4B, its the same model that the wood folk were losing their chips over a few days ago.First, its important to understand it is a fully local model.
No internet connectivity is necessary.How to use it:
1. Download Edge gallery from Play Store
2. Go to models and pick Gemma-4-E4BYou may want to do it on WiFi as its 3.5 GB.
Then just run it via Edge gallery.
No #datacentre
No gas turbines
No rivers for coolingOn a aging 8 Core cell, with 32GB of ram and a GPU it runs very well speed wise.
Its "only" 4 billion parameters, so how good is it?
1/4
-
#LocalLLM update! I was able to implement this rather complicated feature using Pi and Qwen3.6 27B.
It added a rendering state to my #Chit LLM chat app that renders rectangles over the uploaded image.
Basically you ask for a specific JSON format back, and if it detects it in the stream, it'll create a new image, rendering the rectangles over the objects.
It didn't work immediately out of the gate, but it took a couple turns to fix some bugs. But ultimately it DID work, and so far it works great. :D
But the real headline is that I didn't use Claude or OpenAI stuff. All local. 4090. 24gb.
That's the future we need to head towards if we're going to be using AI for coding going forward. If we're going to shift towards this method of development, we CANNOT be bound to any company, with their leash around our necks and wallets.
-
#LocalLLM update! I was able to implement this rather complicated feature using Pi and Qwen3.6 27B.
It added a rendering state to my #Chit LLM chat app that renders rectangles over the uploaded image.
Basically you ask for a specific JSON format back, and if it detects it in the stream, it'll create a new image, rendering the rectangles over the objects.
It didn't work immediately out of the gate, but it took a couple turns to fix some bugs. But ultimately it DID work, and so far it works great. :D
But the real headline is that I didn't use Claude or OpenAI stuff. All local. 4090. 24gb.
That's the future we need to head towards if we're going to be using AI for coding going forward. If we're going to shift towards this method of development, we CANNOT be bound to any company, with their leash around our necks and wallets.
-
#LocalLLM update! I was able to implement this rather complicated feature using Pi and Qwen3.6 27B.
It added a rendering state to my #Chit LLM chat app that renders rectangles over the uploaded image.
Basically you ask for a specific JSON format back, and if it detects it in the stream, it'll create a new image, rendering the rectangles over the objects.
It didn't work immediately out of the gate, but it took a couple turns to fix some bugs. But ultimately it DID work, and so far it works great. :D
But the real headline is that I didn't use Claude or OpenAI stuff. All local. 4090. 24gb.
That's the future we need to head towards if we're going to be using AI for coding going forward. If we're going to shift towards this method of development, we CANNOT be bound to any company, with their leash around our necks and wallets.
-
#LocalLLM update! I was able to implement this rather complicated feature using Pi and Qwen3.6 27B.
It added a rendering state to my #Chit LLM chat app that renders rectangles over the uploaded image.
Basically you ask for a specific JSON format back, and if it detects it in the stream, it'll create a new image, rendering the rectangles over the objects.
It didn't work immediately out of the gate, but it took a couple turns to fix some bugs. But ultimately it DID work, and so far it works great. :D
But the real headline is that I didn't use Claude or OpenAI stuff. All local. 4090. 24gb.
That's the future we need to head towards if we're going to be using AI for coding going forward. If we're going to shift towards this method of development, we CANNOT be bound to any company, with their leash around our necks and wallets.
-
#LocalLLM update! I was able to implement this rather complicated feature using Pi and Qwen3.6 27B.
It added a rendering state to my #Chit LLM chat app that renders rectangles over the uploaded image.
Basically you ask for a specific JSON format back, and if it detects it in the stream, it'll create a new image, rendering the rectangles over the objects.
It didn't work immediately out of the gate, but it took a couple turns to fix some bugs. But ultimately it DID work, and so far it works great. :D
But the real headline is that I didn't use Claude or OpenAI stuff. All local. 4090. 24gb.
That's the future we need to head towards if we're going to be using AI for coding going forward. If we're going to shift towards this method of development, we CANNOT be bound to any company, with their leash around our necks and wallets.
-
Is anybody using a Strix Halo machine with 128GB uRAM?
I have a couple of questions...
-
Is anybody using a Strix Halo machine with 128GB uRAM?
I have a couple of questions...
-
Is anybody using a Strix Halo machine with 128GB uRAM?
I have a couple of questions...
-
Is anybody using a Strix Halo machine with 128GB uRAM?
I have a couple of questions...
-
Is anybody using a Strix Halo machine with 128GB uRAM?
I have a couple of questions...
-
Find the best local LLM for your hardware, ranked by benchmarks
https://github.com/Andyyyy64/whichllm
#HackerNews #localLLM #hardware #benchmarks #AItools #machinelearning #GitHub
-
Find the best local LLM for your hardware, ranked by benchmarks
https://github.com/Andyyyy64/whichllm
#HackerNews #localLLM #hardware #benchmarks #AItools #machinelearning #GitHub
-
Find the best local LLM for your hardware, ranked by benchmarks
https://github.com/Andyyyy64/whichllm
#HackerNews #localLLM #hardware #benchmarks #AItools #machinelearning #GitHub
-
Find the best local LLM for your hardware, ranked by benchmarks
https://github.com/Andyyyy64/whichllm
#HackerNews #localLLM #hardware #benchmarks #AItools #machinelearning #GitHub
-
Find the best local LLM for your hardware, ranked by benchmarks
https://github.com/Andyyyy64/whichllm
#HackerNews #localLLM #hardware #benchmarks #AItools #machinelearning #GitHub