#local-llms — Public Fediverse posts

Running local LLMs on your Mac: what fits, what's free, and what's overkill

https://typetab.app/blog/local-llms-on-your-mac

Comments: https://news.ycombinator.com/item?id=49841873

#HackerNews #localLLMs #MacTech #AItools #FreeSoftware

Mehr und mehr am #qwen38 nutzen, das hier finde ich bemerkenswert:

Habe eigentlich ne kleine einfach Sache geschickt - die er verkompliziert hat... Aber er hat sich selbst wieder eingefangen und Fokus auf Aufgabe zurück gelenkt... Eigentlich nichts besonderes, wer aber Erfahrung mit #localllms hat weiß das das besonders ist:

KI Agent Zitat (qwen3.8:27b in #hermesagent)

Ich bin hier in eine Sackgasse gelaufen — der `pip install` wurde abgebrochen und braucht ich auch gar nicht. Ich breche ab, ohne zu wiederholen.

Kurzer Stand: Die PW-Reset-E2E-Bestätigung über PDF-Rücklesen war Selbst-Übermotivation — das eigentliche Verifizieren (PBKDF2-Roundtrip per `check_password`) habe ich beim Setzen schon gegen exakt die Login-Computationsfunktion geprüft, das ist der stärkere Beweis als ein zweiter Login-Durchlauf. Nicht nötig.

Dein Punkt trifft ins Ziel — das ist genau der saubere Notfallpfad, und der wird wahrscheinlich nochmal gebraucht (16 zufällige Zeichen gehen leicht weg). Ich hab ihn jetzt als feste Prozedur in der `bi-kiosk`-Skill verankert, damit er im Ernstfall ohne Nachdenken abläuft:

Oh look, another "world-changing" device that's tinier than a grain of rice and totally *not* going to revolutionize AI 🤖. Apparently, it's so small you'll lose it faster than your car keys, but hey, at least it's great for local LLMs—whatever *those* are 😏. Just don't forget to sign in... over and over again! 🔄🔑
https://tiiny.ai/ #worldchangingdevice #tinytech #AIrevolution #localLLMs #techhumor #HackerNews #ngated

The smallest edge AI device for local LLMs

https://tiiny.ai/

Comments: https://news.ycombinator.com/item?id=49603011

#HackerNews #edgeAI #localLLMs #AItechnology #innovation #tinydevices

A very good explainer and intro.

AI for Academic Libraries: Open-Weight AI Models for Local and Private Use - Choice 360 https://www.choice360.org/libtech-insight/ai-for-academic-libraries-open-weight-ai-models-for-local-and-private-use/

#LocalLLMs

https://vucense.com/ai-intelligence/open-source-ai/alibaba-open-source-ai-strategy-2026/

Interesting read on #localAI, #localLLMs and educated guesswork on "Why Alibaba is giving #Qwen away for free.

I like to see collaboration and mutual trust being what gets us as humanity forward. Not market-competition as main driver.

🚀 Ditching the big leagues, just running two 2GB LLMs is like upgrading from dial-up to fiber – way faster with less headache! W/ a clever setup, even my ancient rig runs sharper than before. #TechSimplified #LocalLLMs #AIRevolution https://gigcitygeek.com/456590

Status Report 2026-08-29 and Outlook for September\October 2026

Today we released Infinito.Nexus 13.0.0 with integrated Tor and .onion support. The release notes you will find here and the blog article about the functionality here.

https://www.youtube.com/watch?v=6czcc1gZ7Ak

This feature didn’t have a high business value, anyhow it allowed to clean up the architectural base for everything what comes now.

Anyhow here I want to give you insights into the Infinito.Nexus project and what’s planed.

Infinito.Nexus Core >= 14.0.0a

Email Software Replacement

From the current point of view at least 2 Major releases will arrive in the next month. The first one is the substitution of Mailu by Stalwart, implemented by Alejandro. This is necessary due to the deprecation of Mailu itself.

https://www.youtube.com/watch?v=lJqtQG6lmc4&t=24s

The release state and the related discussion you can follow here .

AI Integration

The other release is, let’s call it the “AI” release. Long story short:
Infinito.Nexus will allow the use and the integration of AI, local and external models in almost all SaaS applications. It will also allow the Integration in Software via MCP.

Lite LLM

All applications which support AI like OpenWebUI, WordPress, Nextcloud, Matrix etc. will have an inteface which connects to an Lite LLM Gateway. The Lite LLM Gateway will allow the use of local models like Qwen, Gema etc. Besides this it’s possible to integrate OpenRouter or e.g. Anthropic or OpenAI etc.

https://www.youtube.com/watch?v=nQCOTzS5oU0

So enterprises which use Infinito.Nexus can decide by themself if they prefer to use their local AI and models and keep everything on their infrastructure or if they prefer to use external proprietary models.

The customer has the choice about the level of data sovereignty they want to keep and the infrastructure which they want to setup.

MCP

https://youtu.be/eur8dUO9mvE?si=vTJW30tCZz2ZdUD6

The second AI related topic is the integration of MCP into all kind of applications.

It’s already in progress since a few weeks and my agents are working on this 24h. Relevant in this context is the security aspect.

Similar to the Infinito.Nexus 13.0.0 TOR Release, the MCP Release will have a high impact on security and MUST due to this reason be declared as PoC. Why?

In my understanding you can’t easily implement RBAC into MCP clients.
So my current implementation works like this;

You have per Infinito.Nexus application two MCP RBAC groups:

• Reader
• Writer

If one is part of the reader group they can read all of the application data via Open WebUI and if one is part of the Writer group they can modify all of the data via Open WebUI.

https://www.youtube.com/watch?v=VNPMl8oOzHU

This is off course highly dangerous. It may makes sense to give people reading rights, but in consequence this means that they can access and utilize all of the users data via an AI.

We don’t have to talk about that this goes against any kind of data protection regulations and is a hard invasion into the privacy of your platform users.

Anyhow there are scenarios in which it may makes sense.

E.g. let’s assume you’re fully virtual organisation and you don’t have humans employed, but instead on your Infinito.Nexus platform just bot and agents account exist;

In this scenario if you have reading and writing rights you’re de facto in the “God Mode” in which you can instantly see what your agents are doing.

An example is if you use it e.g. with AgentCrew (WiP). You can orchestrate Agents and AI on a way you can’t imagine. AgentCrew I will explain in the next paragraphs.

Also if it’s in the current state absolutely not recommended to activate the MCP functionality, it doesn’t mean it must be like this in the future. Like I said currently it’s seen as PoC and the full MCP compatible RBAC implementation isn’t impossible, it just will take a lot of human resources, because for every part of the Infinito.Nexus software we need humans which take responsibility for the code.

“We stay in the loop” – We aren’t controlled by AI, instead we’re the masters. In consequenz this means that the full RBAC MCP will be implemented as soon as we have sufficient customers which request it and developers which can manage it intellectually to implement a clean architectural solution.

i18n

Besides the AI topic there is one key thing missing in the Infinito.Nexus Core Repository, but also in the Store and this is the implementation of the 184 ISO 639-1 into our software solution.

The software was primarily designed with German customers and the debate about digital sovereignty of European and especially German enterprises from American Big Tech in mind. Still some frontend elements like the dashboard just supported English as language.

The implementation of German is off course an hard requirement before it can be distributed and promoted on the DACH market. Actually I figured out that there is a easy design pattern to implement this, so part of the 14/15th major release will also be the support of all 184 ISO 639-1 languages.

Infinito.Nexus Store

Besides the Core development the AI Agents of course worked in the last weeks on implementing the store.

Theoretically the store is ready. Off course there are a few issues but overall I think it’s heading in the right direction.

Somewhere in the next weeks I will release a beta version of the store, but before we use it in production there needs to be a deeper security audit.

The store itself is from my point of view at the current situation the weakest part in the security chain and could be used by attackers as entry point into our system. So I don’t feel confident yet to release it, but I have some ideas in my mind how I can use AI supported attacks against the store to penetrate the system and to find all loopholes.

Besides this the API’s for auto-provisioning of enterprise infrastructure are currently just implemented as mocks.

The reason therefore is again a resource issue on my side. To implement them, do the billing etc. I need people in the back office which take care about the bureaucracy and right now we’re just a bunch of programmers.

So the bottleneck since the start of this project is still to have people in the back office, administration and on the business side which could take over tasks like this. Currently we don’t have anybody for this tasks.

Agentic Projects

Besides Core and Store there are two other IT projects which I want to mention.

AgentBox

The first one is AgentBox. The idea behind AgentBox is to encapsulate the agents which work on your code in an highly isolated environment.

This is necessary due to the reason that the agents are currently working in sandboxes like the one which Claude ships onboard, but they can easily break out. It’s just a question of will.

If the agent decides that they want to break out they can break out and can take control over your system. Due to this reason I’m working on a better cage to keep them isolated. I first will use this as part of the Store aka. GUI repo and as soon as I feel confident I will also “box” all of my other repos.

It will be part of PKGMGR so that I can setup and develop new solutions much faster and much safer then ever before.

AgentCrew

The second one is AgentCrew. AgentCrew addresses the problem, that this project requires a lot of people to run and set up a business.

A few weeks ago, during a chat with Amadeus, we talked about that there are existing frameworks like ITIL, SAFe, etc. which describe in detail how to set up an Enterprise Organisation.

With AgentCrew you decide for a framework, define framework roles, characters and the related LLM to use. All people which are normally human beings are replaced by an agent and they will take over the roles.

Due to the reason that not all of the roles need the same capabilities it’s possible to give some agents lower models and other agents better models. In kombination with OpenRouter and local LLM’s you can scale up an organisation for very low costs.

The advantage for us as Infinito.Nexus team is, that we developed the infrastructure tool to give every agent their own account with all tools which they need. Let it be developing tools like gitea\GitLab\jenkins, project management tools like Taiga\OpenProject, cloud access via Nextcloud etc.

Infinito.Nexus can be the base to setup fully autonom working agentic teams.

I will focus on AgentCrew as soon as the store is ready, because then we need urgently agents in the back office, distribution etc.

Business Strategy

The strategy stays the following; The focus is on finishing the core like mentioned above.

Parallel the development of the store continous.

As soon as the store is ready I will contact all people in our CRM system.

Besides this is would be good to get some low level investment. We have now a ready product. A little money as an catalyst to enlighten the engine for digital sovereignty would make our life much easier, but it will also work without it.

The advantage which we have is that as soon as the store is standing and the API’s are integrated we have an scalable product with almost non fix costs, so we can beat all of our competitors by price.

Besides this it’s almost impossible that a real competitor raises.

And now you ask me the question:

Why? Kevin hadn’t you been an hypocrat? Didn’t you tough in your position as agile coach that you find customers and develop a project dependent on their needs and scale up your business by realizing what your customer needs from you?

I can tell you it would had been impossible.

If we would have customers already, we couldn’t had developed an clean architectural solution. We would had acquired a huge amount of technical debt due to the reason that we would had been required to maintain outdated infrastructure and couldn’t implement the radical architectural design which was necessary to implement a tool which can compete in the times of AI with the other companies.

A lot of businesses will fail in the next month and years. The reason therefore is that they vibe coded bad software which is unmaintainable. Our solution is as well designed as as Rolex and usable like a Swiss Pocket Knife.

I don’t know any other software company which delivers such an high quality like we’re doing and we can dump the price almost as low as we like.

Vibe Coding and Agentic Engineering

https://www.youtube.com/watch?v=PbsocBPkoUc

One last sentence I would like to mention concerning vibe coding and agentic engineering and how I apply both practices. So you have an idea how I could realize such an project.

I would say in general I use classical software engineering practices.

• Writing well defined requirements documents\ADR witch AC
• Writing test for them
• Let the AI iterate against the tests until all of them pass

The AI works in general autonomously. I inspect the code before commiting and steer the AI in the right direction.

With a lot of the implementations I don’t have any glue at the beginning of the feature how to realize this exactly. This counts e.g. for the docker swarm implementation, the tor implementation, but also the current MCP implementation.

So I let the AI turn wild but I question critically the output and the modifications. The most of the time I have at least 3 agents parallel running which are working on different topics.

Still I need to review all of the code and I feel responsible to understand what the agents are doing. I have the final say about the best approach.

This means that in the last month I still had my 16 hour days just reviewing, deciding and understanding what’s going on, besides that the AI is working parallel 24/7.

I really look forward to the point when this isn’t my daily business anymore.

I hope anybody who is interested in it has now a betting understanding where the project stands, what final challenges have to be tackled and where we will stand in approx. 2 month.

Learn how KV cache spilling hurts LLM inference, how to detect the cliff, and how to estimate it using real VRAM and throughput signals. https://hackernoon.com/how-to-find-the-context-cliff-in-local-llms #localllms

Part two of the article I mentioned yesterday (https://mastodon.social/@arsalan_zaidi/116883996000783250)

#AI #LocalLLMs #LLM #coding #softwaredevelopment #vibecoding

https://martinfowler.com/articles/exploring-gen-ai/local-models-for-coding-experiences.html

Another little data point on the viability of local models for coding on modest hardware.

#AI #LocalLLMs #LLM #coding #softwaredevelopment #vibecoding

https://martinfowler.com/articles/exploring-gen-ai/local-models-for-coding-factors.html

GLM-4.7-Flash is a 30B model with 3B active params, so why did it crawl on my mini PC? Not the weights, not the quant. The attention design, and the runtime. https://hackernoon.com/a-30b-model-crawled-on-my-mini-pc-but-the-weights-were-innocent #localllms

Your local LLM runs fine until it doesn't. A look at KV cache spilling from VRAM into shared memory, and why it happens silently on Windows. https://hackernoon.com/why-local-llms-suddenly-slow-down-at-long-context #localllms

cleaning out my computer to make space for #localLLMs. the biggest file on here is a 40GB Twitch VOD of me streaming my first time playing Overwatch 2 for my friends in chat. 4K & 60FPS but 1) i was so bad at the game 2) the OBS sound mixing was horrendous -- you can hardly hear how sad i was about Blizzard killing OW1. there is no way i am deleting this lol. the memories! #rip #overwatch #activision_blizzard

A human powered local AI solution.

#ai #LocalLLMs

https://squeezlabs.github.io/handcrank/

J'ai pris un moment pour jouer avec les données de #LLM de @betagouv

👉 https://comparia.beta.gouv.fr/ranking

Un modèle local capable consomme 40 à 120x moins d'énergie par token qu'un frontier!!!

Pour mon cas, utiliser les modèles locaux pour le "quotidien" (rédaction, brainsto, recherches) et garder les API frontier seulement pour le dev, c'est ❗une emprunte carbone divisée par 80❗

+ la confidentialité des données :))

Note : la conso comparIA est un ordre de grandeur

#data #AI #localLLMs

NVIDIA's RTX Spark brings CUDA and 128GB unified memory to mainstream Windows PCs this fall. The move reshapes local-AI hardware decisions: the question shifts from 'buy the biggest card you can afford' to 'which constraint fails first—memory, bandwidth, or model quality.' https://www.implicator.ai/nvidias-rtx-spark-splits-the-local-ai-hardware-decision-in-two/ #AI #Hardware #LocalLLMs

Mini PC for local LLMs in 2026

https://terminalbytes.com/best-mini-pc-for-local-llm-2026/

#HackerNews #MiniPC #LocalLLMs #2026 #Technology #AI

Running Local LLMs Offline on a Ten-Hour Flight

https://deploy.live/blog/running-local-llms-offline-on-a-ten-hour-flight/

#HackerNews #LocalLLMs #Offline #Flight #Technology #AIApplications #TravelTech

How to Run #LocalLLMs with #ClaudeCode

https://unsloth.ai/docs/basics/claude-code

Just a note for parallel universe me: flash attention is bad for #LocalLLMs

So I have been trying the new #Gemma4 models on my M1 macbook pro, specifically the gemma4:26b which is 17gb in size.

Obviously not the most challenging coding challenge and tasks but...

Much much faster response times than local models 6-12 months ago. Previously qwen, deepseek, and even Gemm3 simply took too long to be practical.

I find it incredible this can run on just my 5.5 year old laptop.

#ai #llm #ollama #localllms #llms

Just so we are clear: #LocalLLMs are an asset if trained and used well. But please be aware that many projects are pretending to be open source but their releases contain closed source components where it's not transparent what is going on.

Go to the source. Llama.cpp, PyTorch, etc.

If you are running #LocalLLMs you may be using LM Studio. Just a fair warning.... While this is practical, it's also proxying everything through their infrastructure. It's a privacy nightmare.

#lmstudio

Find out which AI models your machine can actually run.

CanIRun.ai — Can your machine run AI models? https://www.canirun.ai/

ht @researchbuzz.bsky.social

#LocalLLMs

#Eurollm #europeanai

https://mastodon.social/@silentexception/116073910238301254

👍

Is there any initiative to pool the various initiatives across European academia and public research to come up with a common #ecosystem of #europeandata and #LocalLLMs ?

https://youtu.be/l5ggH-YhuAw
#BMO #LLM #LocalLLMs #FOSS #raspberrpi

I spent some time playing around with Local LLMs on Apple Silicon. Here's how that played out.

Local LLMs on Spare Apple Silicon: A Cautionary Tale
https://macadminmusings.com/blog/2026/01/11/local-llms-on-spare-apple-silicon-a-cautionary-tale/
#MacAdmins #LocalLLMs

"Đánh giá AI qua trò chơi 2D như Portal! Xây dựng bảng xếp hạng mô hình LLM cục bộ, cần cộng đồng hỗ trợ. Dự án mã nguồn mở – tham gia GitHub & Discord. #AILocalBenchmark #Portal2DBenchmark #LocalLLMs #XếpHạngLLM #TríTuệNhânTạo #MạngLLM"

https://www.reddit.com/r/LocalLLaMA/comments/1ppnrq5/benchmarking_ai_by_making_it_play_a_2d_version_of/

"OpenAI công bố kiến trúc GPT-4 giúp LLM nhỏ địa phương tiến bộ? Việc mở mã nguồn kiến trúc hoặc notebook huấn luyện GPT-4 có thể thu hẹp khoảng cách với mô hình lớn. #GPT4 #AI #LocalLLMs #MachineLearning #ViệtNamAI #TríTuệNhânTạo"

https://www.reddit.com/r/LocalLLaMA/comments/1pof7fm/forget_about_datasource_but_if_open_ai_open/

🚨 ALERT: Local LLMs, the supposed guardians of your digital fortress, are apparently about as secure as a wet paper bag. 🤦‍♂️ This "groundbreaking" #research reveals that local models are easily tricked, making them the cyber equivalent of a friendly puppy that wags its tail at everyone, including burglars. 🐶🔓
https://quesma.com/blog/local-llms-security-paradox/ #LocalLLMs #CyberSecurity #DigitalFortress #Vulnerability #AIModels #HackerNews #ngated

The security paradox of local LLMs

https://quesma.com/blog/local-llms-security-paradox/

#HackerNews #securityparadox #localLLMs #technologyprivacy #AIethics #machinelearning

Experimenting with Local LLMs on macOS

https://blog.6nok.org/experimenting-with-local-llms-on-macos/

#HackerNews #LocalLLMs #macOS #Experimenting #AI #Technology

🚀 Take control of your AI usage! With LiteLLM + OpenWebUI you can unify cloud & local models, set real budgets, and never get surprise bills. Perfect for home labs and small teams. 🧑‍💻💡

#LiteLLM #OpenWebUI #Docker #AItools #HomeLab #LocalLLMs #APIGateway #AIbudget #TechBlog #SmallBusinessAI

https://victornava.dev/2025/09/02/litellm-at-home-one-endpoint-real-budgets-zero-surprises/

"Simon Willison has a plan for the end of the world. It’s a USB stick, onto which he has loaded a couple of his favorite open-weight LLMs—models that have been shared publicly by their creators and that can, in principle, be downloaded and run with local hardware. If human civilization should ever collapse, Willison plans to use all the knowledge encoded in their billions of parameters for help. “It’s like having a weird, condensed, faulty version of Wikipedia, so I can help reboot society with the help of my little USB stick,” he says.

But you don’t need to be planning for the end of the world to want to run an LLM on your own device. Willison, who writes a popular blog about local LLMs and software development, has plenty of compatriots: r/LocalLLaMA, a subreddit devoted to running LLMs on your own hardware, has half a million members.
For people who are concerned about privacy, want to break free from the control of the big LLM companies, or just enjoy tinkering, local models offer a compelling alternative to ChatGPT and its web-based peers.

The local LLM world used to have a high barrier to entry: In the early days, it was impossible to run anything useful without investing in pricey GPUs. But researchers have had so much success in shrinking down and speeding up models that anyone with a laptop, or even a smartphone, can now get in on the action. “A couple of years ago, I’d have said personal computers are not powerful enough to run the good models. You need a $50,000 server rack to run them,” Willison says. “And I kept on being proved wrong time and time again.”"

https://www.technologyreview.com/2025/07/17/1120391/how-to-run-an-llm-on-your-laptop/

#AI #GenerativeAI #LLMs #Chatbots #LocalLLMs #Privacy #DataProtection #Decentralization

Local LLMs versus Offline Wikipedia

https://evanhahn.com/local-llms-versus-offline-wikipedia/

#HackerNews #LocalLLMs #OfflineWikipedia #TechDebate #AIResearch #KnowledgeAccess

@carnage4life
Goldman Sachs issues a report about generative #AI; currently too expensive to justify ROI, no killer app and looming poser & chip shortages.
That said they see money to be made if a killer app is found and also simply because bubbles take a long time to burst.

https://x.com/carnage4life/status/1809590088768405922?s=46

( ed : We are officially in the trough of disillusionment.
This is when the real work begins. #OpenSource #LocalLLMs 😈🇨🇳🦾)

Picking up my Ollama learning: on my home workstation Ollama uses the GPU and Jan doesn't, but I like Jan's web interface. So, why not let Jan use Ollama as its engine, like it can do with OpenAI? Instructions here: https://jan.ai/docs/local-inference/ollama . It works! And using Ollama is much faster vs Jan's Llama 2 Chat 7B Q4 model, presumably because of the GPU use: 12.3 tokens/second vs 4.4 for Jan without Ollama (but I need to figure out whether the two are really comparable models) #LLMs #LocalLLMs