#local-llms — Public Fediverse posts

Learn how KV cache spilling hurts LLM inference, how to detect the cliff, and how to estimate it using real VRAM and throughput signals. https://hackernoon.com/how-to-find-the-context-cliff-in-local-llms #localllms

Part two of the article I mentioned yesterday (https://mastodon.social/@arsalan_zaidi/116883996000783250)

#AI #LocalLLMs #LLM #coding #softwaredevelopment #vibecoding

https://martinfowler.com/articles/exploring-gen-ai/local-models-for-coding-experiences.html

Another little data point on the viability of local models for coding on modest hardware.

#AI #LocalLLMs #LLM #coding #softwaredevelopment #vibecoding

https://martinfowler.com/articles/exploring-gen-ai/local-models-for-coding-factors.html

GLM-4.7-Flash is a 30B model with 3B active params, so why did it crawl on my mini PC? Not the weights, not the quant. The attention design, and the runtime. https://hackernoon.com/a-30b-model-crawled-on-my-mini-pc-but-the-weights-were-innocent #localllms

Your local LLM runs fine until it doesn't. A look at KV cache spilling from VRAM into shared memory, and why it happens silently on Windows. https://hackernoon.com/why-local-llms-suddenly-slow-down-at-long-context #localllms

A human powered local AI solution.

#ai #LocalLLMs

https://squeezlabs.github.io/handcrank/

J'ai pris un moment pour jouer avec les données de #LLM de @betagouv

👉 https://comparia.beta.gouv.fr/ranking

Un modèle local capable consomme 40 à 120x moins d'énergie par token qu'un frontier!!!

Pour mon cas, utiliser les modèles locaux pour le "quotidien" (rédaction, brainsto, recherches) et garder les API frontier seulement pour le dev, c'est ❗une emprunte carbone divisée par 80❗

+ la confidentialité des données :))

Note : la conso comparIA est un ordre de grandeur

#data #AI #localLLMs

NVIDIA's RTX Spark brings CUDA and 128GB unified memory to mainstream Windows PCs this fall. The move reshapes local-AI hardware decisions: the question shifts from 'buy the biggest card you can afford' to 'which constraint fails first—memory, bandwidth, or model quality.' https://www.implicator.ai/nvidias-rtx-spark-splits-the-local-ai-hardware-decision-in-two/ #AI #Hardware #LocalLLMs

Mein #arbeitgeber labert grade in so nem #MicrosoftTeams Call für alle Mitarbeiter was von #digitalesouveranitat und dann soll ich MEHR mit #microsoft #github #copilot machen. Und selbstverständlich wird ALLES #ai. Sogar unsere TLD wechselt von .net auf .ai.
Wir sollen ganz explizit doch bitte #ki in die tägliche #Arbeit einbinden, der Vertrieb soll sich beim. Aber bloß "unsere" nutzen, wegen den Daten. Muss mich gleich mal informieren, ob #localLLMs erlaubt sind.
https://hessen.social/@Moonstone2487/116082676681677166

Mini PC for local LLMs in 2026

https://terminalbytes.com/best-mini-pc-for-local-llm-2026/

#HackerNews #MiniPC #LocalLLMs #2026 #Technology #AI

Running Local LLMs Offline on a Ten-Hour Flight

https://deploy.live/blog/running-local-llms-offline-on-a-ten-hour-flight/

#HackerNews #LocalLLMs #Offline #Flight #Technology #AIApplications #TravelTech

How to Run #LocalLLMs with #ClaudeCode

https://unsloth.ai/docs/basics/claude-code

Just a note for parallel universe me: flash attention is bad for #LocalLLMs

So I have been trying the new #Gemma4 models on my M1 macbook pro, specifically the gemma4:26b which is 17gb in size.

Obviously not the most challenging coding challenge and tasks but...

Much much faster response times than local models 6-12 months ago. Previously qwen, deepseek, and even Gemm3 simply took too long to be practical.

I find it incredible this can run on just my 5.5 year old laptop.

#ai #llm #ollama #localllms #llms

Just so we are clear: #LocalLLMs are an asset if trained and used well. But please be aware that many projects are pretending to be open source but their releases contain closed source components where it's not transparent what is going on.

Go to the source. Llama.cpp, PyTorch, etc.

If you are running #LocalLLMs you may be using LM Studio. Just a fair warning.... While this is practical, it's also proxying everything through their infrastructure. It's a privacy nightmare.

#lmstudio

Find out which AI models your machine can actually run.

CanIRun.ai — Can your machine run AI models? https://www.canirun.ai/

ht @researchbuzz.bsky.social

#LocalLLMs

#Eurollm #europeanai

https://mastodon.social/@silentexception/116073910238301254

👍

Is there any initiative to pool the various initiatives across European academia and public research to come up with a common #ecosystem of #europeandata and #LocalLLMs ?

I spent some time playing around with Local LLMs on Apple Silicon. Here's how that played out.

Local LLMs on Spare Apple Silicon: A Cautionary Tale
https://macadminmusings.com/blog/2026/01/11/local-llms-on-spare-apple-silicon-a-cautionary-tale/
#MacAdmins #LocalLLMs

🚨 ALERT: Local LLMs, the supposed guardians of your digital fortress, are apparently about as secure as a wet paper bag. 🤦‍♂️ This "groundbreaking" #research reveals that local models are easily tricked, making them the cyber equivalent of a friendly puppy that wags its tail at everyone, including burglars. 🐶🔓
https://quesma.com/blog/local-llms-security-paradox/ #LocalLLMs #CyberSecurity #DigitalFortress #Vulnerability #AIModels #HackerNews #ngated

The security paradox of local LLMs

https://quesma.com/blog/local-llms-security-paradox/

#HackerNews #securityparadox #localLLMs #technologyprivacy #AIethics #machinelearning

Experimenting with Local LLMs on macOS

https://blog.6nok.org/experimenting-with-local-llms-on-macos/

#HackerNews #LocalLLMs #macOS #Experimenting #AI #Technology

🚀 Take control of your AI usage! With LiteLLM + OpenWebUI you can unify cloud & local models, set real budgets, and never get surprise bills. Perfect for home labs and small teams. 🧑‍💻💡

#LiteLLM #OpenWebUI #Docker #AItools #HomeLab #LocalLLMs #APIGateway #AIbudget #TechBlog #SmallBusinessAI

https://victornava.dev/2025/09/02/litellm-at-home-one-endpoint-real-budgets-zero-surprises/