#llmops — Public Fediverse posts

Most LLM eval dashboards are decoration.

The test for any metric: if this number moved, what would you do differently? No answer means it is a vanity metric and you are paying tokens to compute it.

The regressions that actually hurt are silent. A provider updates a model version, a retrieval index grows and starts returning different documents, and nobody knows until a customer complains days later.

https://www.adaptiverecall.com/llm-evaluation/

#LLMOps #AIEngineering

📊 Prometheus metrics cover requests, errors, latency, tokens and routing overhead

⚠️ Apache 2.0, pre-alpha and evolving fast — the API is expected to change before v1.0 and it is explicitly not meant for production yet

🌐 https://github.com/NVIDIA-NeMo/Switchyard
#LLMOps #inference

Running Claude Code on Fable or Opus 5 and wondering where your tokens go? It might be your subagents.

Claude Code picks a subagent's model in this order: env var, explicit dispatch parameter, agent frontmatter, session model. The pin everybody uses is rank 3 of 4, and it has silently dropped out across several releases. No error, just a bigger number four weeks later.

These few lines of code fix it: https://thomas-witt.com/blog/blog-subagent-model-pin/

#ClaudeCode #LLMOps #DevTools #Bash #AI

LLM-судье нельзя верить на слово: как построить надёжный гейт и проверить сами тесты

Как перестать доверять LLM-судье на слово и построить безопасную двухконтурную систему оценки? Внутри статьи: Архитектурный паттерн сдерживания: почему у классической нормализации должно быть право вето. CI-инварианты: как ловить галлюцинации моделей с помощью враждебных фикстур в grounded-judge-gate . Разбор факапов: три реальных бага проектирования, которые едва не увели систему в ложноположительное пике.

https://habr.com/ru/articles/1062614/

#LLM #LLMasajudge #LLMOps #Python #тестирование #валидация_моделей #архитектура_систем #каппа_Коэна #качество_данных

Error logs won't catch a hallucination or a silent quality drop. We traced one agent workflow end to end and watched it run 8 minutes 25 seconds and burn 8,814 tokens in a single call, most of it buried seven levels deep. Tracing with OpenTelemetry is how you actually see inside an LLM app. https://go.upgradejs.com/w2s #LLMOps #Observability #AI

🤖 Agenta-AI/agenta

Builds and runs AI agents with chat-based creation, team sharing and scheduled or event-triggered background execution

⭐ Stars: 4305
📅 Last Update: Jul 19, 2026

https://github.com/Agenta-AI/agenta

#selfhosted #homelab #selfhost #selfhosting #opensource #aiagents #llmops

For agent observability: do you log full prompts/completions, or redact and store hashes + token counts?

Full logs debug incidents fast but become a PII liability and a storage bill. Where's your line — and what do you wish you'd logged from day one?

#AI #Observability #LLMOps

🚀 NEW on We ❤️ Open Source 🚀

Production AI isn't just about better models anymore. It's about building reliable systems around them.

Paul Schweigert and Nathan Fulton show how Mellea provides backend portability, output validation, and efficient local inference for production-ready AI workflows.

https://allthingsopen.org/articles/mellea-small-models-production-ready

#WeLoveOpenSource #OpenSource #LLMOps

I burned my 5-hour Fable 5 token quota in 37 minutes. Here is how I built Suture, a zero-dependency local AI router, to stop the token bleed. https://hackernoon.com/how-i-burned-my-max-tier-quota-in-37-minutes-and-the-zero-dependency-router-i-built-to-fix-it #llmops