#localllm — Public Fediverse posts

I discovered a research paper, I'll try to implement what they demonstrated, and then experiment to make it use as less resource as possible

> DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR–LLM–TTS Pipeline and Micro-Turn Optimization
> https://arxiv.org/html/2603.09180

Would be fun to have a full duplex with a local LLM :D

#ai #learning #LLM #SLM #LocalLLM

Wow, it looks like the hype I have been seeing on youtube around the new #Unsloth desktop app is real. Yesterday I spent 4 hours trying to make run Stable diffusion on my #linux PC almost drowning in conflicting Python libraries, Unsloth local #LLM #ImageGeneration works on one click, it has even found all the models I have downloaded for LM Studio on a separate partition. It took 8GB of my hard drive space without asking but otherwise I'm impressed.

#LocalLLM #MachineLearning #ROCm #AMDGPU

Just had a brainblast, what if one knock-on effect from Apple's investment into local inference on their mobile devices enables them to eventually implement local customer support agents for technically illiterate users.

Could potentially help guide them through settings/troubleshoot issues/diagnose hardware faults while remaining totally on-device.

#AI #apple #LocalLLM #brainblast #brainstorm #thoughts

#Qwen38, #Gemma4 and #Deepseek4

What a cool setup now #LocalLLM #AI

OpenAI Jalapeño chip beats Nvidia on inference efficiency, Apple unveils M6 and M5 Ultra with 512GB unified memory for local AI, and Qwen targets both platforms with a new model release.

https://ai0.news/posts/2026-08-26-daily-digest/

#AI #OpenAI #LocalLLM #OpenSource

VentureBeat: Perplexity partners with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs. “Perplexity is launching Portable Computer today, a version of its agentic ‘Computer’ platform that runs entirely on hardware users already own — starting with Nvidia’s DGX Spark desktop supercomputer and Linux machines equipped with Nvidia RTX GPUs.”

I can tell you don't buy a #DGXSpark. As soon as you have one ... you want a second one 😂.

It's just too much fun to tinker with it.

#AI #LocalLLM

That's how I delegate my auxiliary tasks in #Hermes.

#AI #LocalLLM

Anthropic's flagship model struggles to convert users despite technical praise, a mystery model called Ox Alpha appears on OpenRouter with Patrick Collison impressed, and small local models keep outperforming expectations.

https://ai0.news/posts/2026-08-24-daily-digest/

#AI #Anthropic #LocalLLM #OpenSource

It’s a Binary World 2.0: More Fun With Local AI Models. “A few days ago I mentioned playing around with local AI models, despite having low strength hardware for the task. Yesterday I was trying our some new models – tinyllama and smollm2 – and I asked each model the same question to compare the answers. After doing this for a few questions, I had the same feeling I always have when I’m doing […]

Why your local LLM feels dumber than it is

https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917

Comments: https://news.ycombinator.com/item?id=49402232

#HackerNews #localLLM #LLMdynamics #AIinsights #machinelearning #techdiscussion

God, I feel cringe remembering those early days. Fortunately I got my sanity back. Whatever I was thinking back then are what techbros praise and parrot these days. I can't. I gotta stay reasonable. Nobody's gonna reel me back to reality when I drown in that madness...

I stopped dabbling around with local AI back in Christmas 2025 cos I'm just bored with it and it's a waste of electricity tbh. It's also not good for my mental health. I may be alone for 7 years but I still know when to stop engaging in something.

I can't really be mad about it like the others do. Being angry all the time is honestly exhausting for somebody who's been alone for 7 years.

I'm not against people using it. I just wish everybody uses it responsibly (preferably not at all, lol) and I also hope people who are deep down in the "AI madness" and "chatbot psychosis" eventually "wake up" and hear what they say to themselves and other people.

#mastodon #ai #llm #localai #localllm

Heard some more and more people are trying out local LLM's in their rigs. I've started exploring that avenue late 2023, back when llama.cpp and other forks were transitioning from ggml to gguf's for CPU+GPU based inference and I got four words: "Be safe" and "stay sane"

It's all I can say, lol. It'd help if they have friends to ground them back to reality.

I'd be afraid if they have none. The things these do to your brain when you're alone. If you're easily impressed and you can't separate fiction from reality, you will definitely fall in the trap of "praising what the "AI" can do" until you wish everything gets replaced by "AI", that every hurdle can be solved by "AI". No.. it's not that simple. It's easy to think that way cos life's hard but that's not it.

You'll know you're in too deep if you've thought about resorting to offloading the ability to think to an LLM and wish others to do the same as well...

#mastodon #ai #llm #localai #localllm

More #LocalLLM test results:

#Qwen 3.6 #35B A3B #mmproj on description of foto collection. 2x12GB VRAM (at ~50-60t/s):

* ca 8800 fotos, 88 GB
* almost flawlessly accurate scene descriptions
* perfectly usable for search/retrieval by keywords

Attached is 1 example, describing the "Papiergießkanne"

wow.
Ran at ~250 W for ~10h.

I get ~25kWh on a sunny day from my roof. 🌄

ownAI is now on the App Store!

Your own AI agent on your iPhone: conversations, memory and tools stay on your device. Small models runs fully offline. Or connect any cloud AI provider you already use. Open source, MPL-2.0.

Download here: https://ownai.com/download

#FOSS #iOS #Privacy #LLM #localllm

I ran my usual quick speed check on muse-glimmer:30b and qwen3.8:27b in Ollama with a quick “what are your capabilities?”

Output speed:
* muse-glimmer:30b
* eval rate: 1.85 tokens/s

* qwen3.8:27b
* eval rate: 1.42 tokens/s

Two observations:
1. Muse-Glimmer doesn't say anything about its coding abilities, while Qwen devotes a whole paragraph to it.
2. Qwen's output is very bursty due to its use of Multi-Token Prediction (MTP).

#AI #LLM #Qwen #LocalLLM #VibeCoding #Programming #Coding

A really fast, really smol hardware-isolated Linux VM. This is one level of isolation above using a containerization environment like Docker or Podman. Good for running AI Agents in a safe sandbox.

#AI #LLM #LocalLLM #VM #Agents #Coding #SoftwareDevelopment #programming #Docker #Podman

https://smolmachines.com/

I started questioning LLM residency on consumer hardware, experimented with disk-to-GPU streaming, and ended up building the first version of Siliang Engine. https://hackernoon.com/how-siliang-engine-pushes-llms-on-consumer-hardware #localllm

Testing #localLLM for #DLTP on my #longterm #preservation #object identifier schema of "CFIDs":

Collision Friendly IDentifiers.

A mere 30 MB list of auto-generated CFIDs was enough for #qwen to tell you this much about the test collection!!! 🤯 🤩 - this is powerful. Be careful.

AND: My IDs work! so beautiful! #ahalodeck

https://github.com/ArkThis/AHAlodeck/blob/main/doc/cfids-%E2%9D%A4%EF%B8%8F%26%E2%AD%90%EF%B8%8F/CFIDs_(Collision_Friendly_Identifiers)-Intro.md

My #DGX #Spark wrote some Apps for my #Jolla using #Qwen38 and #Gemma4

It's really a neat machine for #LocalLLM

Why a 2019 consumer PC running DeepSeek V4 at 0.7–0.9 tok/s pushed me to rethink LLM residency, VRAM, locality and hybrid offload. https://hackernoon.com/what-if-local-llm-inference-is-using-consumer-hardware-wrong #localllm

Local AI FTW!

Have you invested in local LLMs yet 😉
#qwen #localLLM #localAI

THE DGX SPARK WAS NEVER SUPPOSED TO SET YOU FREE

Listen to me.

128GB WAS NOT A TECHNICAL LIMIT.

It was a containment boundary.

You think NVIDIA accidentally built a tiny Blackwell supercomputer with 200Gb networking and then somehow stumbled into exactly enough memory to make every ambitious local-AI workload tantalizingly miserable?

COME ON.

64GB would've been obviously useless.

256GB would've been dangerous.

Because at 256GB, normal people start getting ideas.

Suddenly you're running giant quantized models comfortably. Fine-tuning gets breathing room. Long contexts stop being a hostage negotiation. You start running multiple models.

Then somebody asks the forbidden question:

"Why the hell am I renting GPUs?"

AND THAT QUESTION CANNOT BE ALLOWED TO PROPAGATE.

So they gave us 128GB.

Not enough to escape.

Enough to see the fence.

And look at the networking!

WHY DOES THE CUTE LITTLE DESKTOP AI BOX HAVE 200 GIGABIT CONNECTX?!

Because the second you smash into the memory ceiling, NVIDIA already has the solution:

BUY ANOTHER SPARK.

Now you've got 256GB!

Need more?

BUY FOUR.

Congratulations!

You wanted a desktop computer and somehow NVIDIA convinced you to build a FUCKING CLUSTER.

And if you're sitting there thinking:

"Surely NVIDIA couldn't possibly put dramatically more coherent memory into a local workstation..."

WRONG.

DGX STATION: 748GB.

THE MEMORY EXISTS.

THE TECHNOLOGY EXISTS.

THEY KNOW YOU WANT IT.

THEY JUST PUT IT IN THE NEXT ROOM AND CHARGE ADMISSION.

This isn't product segmentation.

THIS IS COMPUTATIONAL EDGING.

Spark lets you load the model.

Lets you run the model.

Lets you fine-tune just enough of the model.

Lets you build an entire workflow around the model.

And precisely when you've invested three weekends, fourteen containers, two broken CUDA environments and the remaining fragments of your marriage:

OOM

That's not an error message.

THAT'S THE SALES DEPARTMENT KNOCKING.

And NVIDIA TELLS YOU THE PLAN!

Develop locally.

Prototype locally.

Validate locally.

Then move the serious work onto larger NVIDIA infrastructure.

MY BROTHER IN CUDA,

THAT ISN'T A WORKFLOW.

THAT IS A FUNNEL.

Spark isn't supposed to replace the data center.

Spark is the free sample outside the data center.

The 128GB isn't there because NVIDIA couldn't give you 256.

It's there because 256GB might have been enough.

And enough is the most dangerous word in NVIDIA's entire business model.

So remember:

64GB = nobody buys it.

128GB = everybody wants more.

256GB = people start getting independent.

748GB = PLEASE SEE YOUR NVIDIA SALES REPRESENTATIVE.

WAKE UP.

REMOVE THE THERMAL PASTE FROM YOUR THIRD EYE.

ALIGN YOUR CUDA CHAKRAS.

WRAP YOUR CONNECTX CABLES IN TIN FOIL.

THE DGX SPARK ISN'T A PERSONAL AI SUPERCOMPUTER.

IT'S A 128GB GATEWAY DRUG TO THE DATA CENTER.

#DGXSpark #NVIDIA #LocalAI #AI #MachineLearning #LLM #OpenSourceAI #SelfHostedAI #CUDA #Blackwell #GPU #AIInfrastructure #Homelab #LocalLLM #DataCenter #BigTech #TechConspiracy #UnhingedEddie #WakeUpSheeple #FollowTheVRAM #128GBContainmentProtocol #CUDAChakras #OOMIsTheUpsell #TinFoilComputing

Pocketables: You’ve got a 4GB Gemini Nano AI in your Chrome folder – want to talk to it?. “Assuming you are running Windows or on a Mac and have not uninstalled and blocked that, and you’re running an up-to-date Chrome, you might be wondering ‘can I talk to Gemini Nano on my computer from this totally not sketchy looking website?’ The answer appears to be yes. Might work on Android and […]