#llmops — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #llmops, aggregated by home.social.
-
Most LLM eval dashboards are decoration.
The test for any metric: if this number moved, what would you do differently? No answer means it is a vanity metric and you are paying tokens to compute it.
The regressions that actually hurt are silent. A provider updates a model version, a retrieval index grows and starts returning different documents, and nobody knows until a customer complains days later.
-
📊 Prometheus metrics cover requests, errors, latency, tokens and routing overhead
⚠️ Apache 2.0, pre-alpha and evolving fast — the API is expected to change before v1.0 and it is explicitly not meant for production yet
🌐 https://github.com/NVIDIA-NeMo/Switchyard
#LLMOps #inference -
📊 Prometheus metrics cover requests, errors, latency, tokens and routing overhead
⚠️ Apache 2.0, pre-alpha and evolving fast — the API is expected to change before v1.0 and it is explicitly not meant for production yet
🌐 https://github.com/NVIDIA-NeMo/Switchyard
#LLMOps #inference -
📊 Prometheus metrics cover requests, errors, latency, tokens and routing overhead
⚠️ Apache 2.0, pre-alpha and evolving fast — the API is expected to change before v1.0 and it is explicitly not meant for production yet
🌐 https://github.com/NVIDIA-NeMo/Switchyard
#LLMOps #inference -
📊 Prometheus metrics cover requests, errors, latency, tokens and routing overhead
⚠️ Apache 2.0, pre-alpha and evolving fast — the API is expected to change before v1.0 and it is explicitly not meant for production yet
🌐 https://github.com/NVIDIA-NeMo/Switchyard
#LLMOps #inference -
📊 Prometheus metrics cover requests, errors, latency, tokens and routing overhead
⚠️ Apache 2.0, pre-alpha and evolving fast — the API is expected to change before v1.0 and it is explicitly not meant for production yet
🌐 https://github.com/NVIDIA-NeMo/Switchyard
#LLMOps #inference -
Running Claude Code on Fable or Opus 5 and wondering where your tokens go? It might be your subagents.
Claude Code picks a subagent's model in this order: env var, explicit dispatch parameter, agent frontmatter, session model. The pin everybody uses is rank 3 of 4, and it has silently dropped out across several releases. No error, just a bigger number four weeks later.
These few lines of code fix it: https://thomas-witt.com/blog/blog-subagent-model-pin/
-
Running Claude Code on Fable or Opus 5 and wondering where your tokens go? It might be your subagents.
Claude Code picks a subagent's model in this order: env var, explicit dispatch parameter, agent frontmatter, session model. The pin everybody uses is rank 3 of 4, and it has silently dropped out across several releases. No error, just a bigger number four weeks later.
These few lines of code fix it: https://thomas-witt.com/blog/blog-subagent-model-pin/
-
Running Claude Code on Fable or Opus 5 and wondering where your tokens go? It might be your subagents.
Claude Code picks a subagent's model in this order: env var, explicit dispatch parameter, agent frontmatter, session model. The pin everybody uses is rank 3 of 4, and it has silently dropped out across several releases. No error, just a bigger number four weeks later.
These few lines of code fix it: https://thomas-witt.com/blog/blog-subagent-model-pin/
-
Running Claude Code on Fable or Opus 5 and wondering where your tokens go? It might be your subagents.
Claude Code picks a subagent's model in this order: env var, explicit dispatch parameter, agent frontmatter, session model. The pin everybody uses is rank 3 of 4, and it has silently dropped out across several releases. No error, just a bigger number four weeks later.
These few lines of code fix it: https://thomas-witt.com/blog/blog-subagent-model-pin/
-
Running Claude Code on Fable or Opus 5 and wondering where your tokens go? It might be your subagents.
Claude Code picks a subagent's model in this order: env var, explicit dispatch parameter, agent frontmatter, session model. The pin everybody uses is rank 3 of 4, and it has silently dropped out across several releases. No error, just a bigger number four weeks later.
These few lines of code fix it: https://thomas-witt.com/blog/blog-subagent-model-pin/
-
LangWatch - AI agent testing and evaluation platform
Cossmology Profile: https://dub.sh/3lc93Cf
Key People: Manouk Draisma, Rogerio Chaves
-
LLM-судье нельзя верить на слово: как построить надёжный гейт и проверить сами тесты
Как перестать доверять LLM-судье на слово и построить безопасную двухконтурную систему оценки? Внутри статьи: Архитектурный паттерн сдерживания: почему у классической нормализации должно быть право вето. CI-инварианты: как ловить галлюцинации моделей с помощью враждебных фикстур в grounded-judge-gate . Разбор факапов: три реальных бага проектирования, которые едва не увели систему в ложноположительное пике.
https://habr.com/ru/articles/1062614/
#LLM #LLMasajudge #LLMOps #Python #тестирование #валидация_моделей #архитектура_систем #каппа_Коэна #качество_данных
-
LLM-судье нельзя верить на слово: как построить надёжный гейт и проверить сами тесты
Как перестать доверять LLM-судье на слово и построить безопасную двухконтурную систему оценки? Внутри статьи: Архитектурный паттерн сдерживания: почему у классической нормализации должно быть право вето. CI-инварианты: как ловить галлюцинации моделей с помощью враждебных фикстур в grounded-judge-gate . Разбор факапов: три реальных бага проектирования, которые едва не увели систему в ложноположительное пике.
https://habr.com/ru/articles/1062614/
#LLM #LLMasajudge #LLMOps #Python #тестирование #валидация_моделей #архитектура_систем #каппа_Коэна #качество_данных
-
LLM-судье нельзя верить на слово: как построить надёжный гейт и проверить сами тесты
Как перестать доверять LLM-судье на слово и построить безопасную двухконтурную систему оценки? Внутри статьи: Архитектурный паттерн сдерживания: почему у классической нормализации должно быть право вето. CI-инварианты: как ловить галлюцинации моделей с помощью враждебных фикстур в grounded-judge-gate . Разбор факапов: три реальных бага проектирования, которые едва не увели систему в ложноположительное пике.
https://habr.com/ru/articles/1062614/
#LLM #LLMasajudge #LLMOps #Python #тестирование #валидация_моделей #архитектура_систем #каппа_Коэна #качество_данных
-
Error logs won't catch a hallucination or a silent quality drop. We traced one agent workflow end to end and watched it run 8 minutes 25 seconds and burn 8,814 tokens in a single call, most of it buried seven levels deep. Tracing with OpenTelemetry is how you actually see inside an LLM app. https://go.upgradejs.com/w2s #LLMOps #Observability #AI
-
Error logs won't catch a hallucination or a silent quality drop. We traced one agent workflow end to end and watched it run 8 minutes 25 seconds and burn 8,814 tokens in a single call, most of it buried seven levels deep. Tracing with OpenTelemetry is how you actually see inside an LLM app. https://go.upgradejs.com/w2s #LLMOps #Observability #AI
-
Error logs won't catch a hallucination or a silent quality drop. We traced one agent workflow end to end and watched it run 8 minutes 25 seconds and burn 8,814 tokens in a single call, most of it buried seven levels deep. Tracing with OpenTelemetry is how you actually see inside an LLM app. https://go.upgradejs.com/w2s #LLMOps #Observability #AI
-
Error logs won't catch a hallucination or a silent quality drop. We traced one agent workflow end to end and watched it run 8 minutes 25 seconds and burn 8,814 tokens in a single call, most of it buried seven levels deep. Tracing with OpenTelemetry is how you actually see inside an LLM app. https://go.upgradejs.com/w2s #LLMOps #Observability #AI
-
小さなLLM（Llama-3.2-1B）をQLoRAでファインチューニングしてFunction Callingを覚えさせてみた
https://qiita.com/Thanush/items/44724e8672f06332ff2e?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items -
小さなLLM（Llama-3.2-1B）をQLoRAでファインチューニングしてFunction Callingを覚えさせてみた
https://qiita.com/Thanush/items/44724e8672f06332ff2e?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items -
🤖 Agenta-AI/agenta
Builds and runs AI agents with chat-based creation, team sharing and scheduled or event-triggered background execution
⭐ Stars: 4305
📅 Last Update: Jul 19, 2026https://github.com/Agenta-AI/agenta
#selfhosted #homelab #selfhost #selfhosting #opensource #aiagents #llmops
-
🤖 Agenta-AI/agenta
Builds and runs AI agents with chat-based creation, team sharing and scheduled or event-triggered background execution
⭐ Stars: 4305
📅 Last Update: Jul 19, 2026https://github.com/Agenta-AI/agenta
#selfhosted #homelab #selfhost #selfhosting #opensource #aiagents #llmops
-
🤖 Agenta-AI/agenta
Builds and runs AI agents with chat-based creation, team sharing and scheduled or event-triggered background execution
⭐ Stars: 4305
📅 Last Update: Jul 19, 2026https://github.com/Agenta-AI/agenta
#selfhosted #homelab #selfhost #selfhosting #opensource #aiagents #llmops
-
For agent observability: do you log full prompts/completions, or redact and store hashes + token counts?
Full logs debug incidents fast but become a PII liability and a storage bill. Where's your line — and what do you wish you'd logged from day one?
-
For agent observability: do you log full prompts/completions, or redact and store hashes + token counts?
Full logs debug incidents fast but become a PII liability and a storage bill. Where's your line — and what do you wish you'd logged from day one?
-
For agent observability: do you log full prompts/completions, or redact and store hashes + token counts?
Full logs debug incidents fast but become a PII liability and a storage bill. Where's your line — and what do you wish you'd logged from day one?
-
For agent observability: do you log full prompts/completions, or redact and store hashes + token counts?
Full logs debug incidents fast but become a PII liability and a storage bill. Where's your line — and what do you wish you'd logged from day one?
-
🚀 NEW on We ❤️ Open Source 🚀
Production AI isn't just about better models anymore. It's about building reliable systems around them.
Paul Schweigert and Nathan Fulton show how Mellea provides backend portability, output validation, and efficient local inference for production-ready AI workflows.
https://allthingsopen.org/articles/mellea-small-models-production-ready
-
🚀 NEW on We ❤️ Open Source 🚀
Production AI isn't just about better models anymore. It's about building reliable systems around them.
Paul Schweigert and Nathan Fulton show how Mellea provides backend portability, output validation, and efficient local inference for production-ready AI workflows.
https://allthingsopen.org/articles/mellea-small-models-production-ready
-
🚀 NEW on We ❤️ Open Source 🚀
Production AI isn't just about better models anymore. It's about building reliable systems around them.
Paul Schweigert and Nathan Fulton show how Mellea provides backend portability, output validation, and efficient local inference for production-ready AI workflows.
https://allthingsopen.org/articles/mellea-small-models-production-ready
-
🚀 NEW on We ❤️ Open Source 🚀
Production AI isn't just about better models anymore. It's about building reliable systems around them.
Paul Schweigert and Nathan Fulton show how Mellea provides backend portability, output validation, and efficient local inference for production-ready AI workflows.
https://allthingsopen.org/articles/mellea-small-models-production-ready
-
🚀 NEW on We ❤️ Open Source 🚀
Production AI isn't just about better models anymore. It's about building reliable systems around them.
Paul Schweigert and Nathan Fulton show how Mellea provides backend portability, output validation, and efficient local inference for production-ready AI workflows.
https://allthingsopen.org/articles/mellea-small-models-production-ready
-
I burned my 5-hour Fable 5 token quota in 37 minutes. Here is how I built Suture, a zero-dependency local AI router, to stop the token bleed. https://hackernoon.com/how-i-burned-my-max-tier-quota-in-37-minutes-and-the-zero-dependency-router-i-built-to-fix-it #llmops
-
I burned my 5-hour Fable 5 token quota in 37 minutes. Here is how I built Suture, a zero-dependency local AI router, to stop the token bleed. https://hackernoon.com/how-i-burned-my-max-tier-quota-in-37-minutes-and-the-zero-dependency-router-i-built-to-fix-it #llmops
-
I burned my 5-hour Fable 5 token quota in 37 minutes. Here is how I built Suture, a zero-dependency local AI router, to stop the token bleed. https://hackernoon.com/how-i-burned-my-max-tier-quota-in-37-minutes-and-the-zero-dependency-router-i-built-to-fix-it #llmops
-
I burned my 5-hour Fable 5 token quota in 37 minutes. Here is how I built Suture, a zero-dependency local AI router, to stop the token bleed. https://hackernoon.com/how-i-burned-my-max-tier-quota-in-37-minutes-and-the-zero-dependency-router-i-built-to-fix-it #llmops
-
I burned my 5-hour Fable 5 token quota in 37 minutes. Here is how I built Suture, a zero-dependency local AI router, to stop the token bleed. https://hackernoon.com/how-i-burned-my-max-tier-quota-in-37-minutes-and-the-zero-dependency-router-i-built-to-fix-it #llmops
-
Your AI agent returned a wrong answer. The logs show it completed successfully. Here is how to build the tracing layer that tells what actually happened & why https://hackernoon.com/tracing-an-ai-agents-reasoning-building-observability-into-your-pipeline #llmops
-
Your AI agent returned a wrong answer. The logs show it completed successfully. Here is how to build the tracing layer that tells what actually happened & why https://hackernoon.com/tracing-an-ai-agents-reasoning-building-observability-into-your-pipeline #llmops
-
RT @Teknium: Ich nehme eine Änderung am Hermes Agent Curator vor, die dazu führt, dass standardmäßig nur ungenutzte Skills bereinigt werden. Die Zusammenführung von Skills erfolgt nicht mehr automatisch, es sei denn, man aktiviert diese Funktion über die Konfiguration oder das Dashboard.
mehr auf Arint.info
#AIConfiguration #CostOptimization #HermesAgent #LLMOps #SkillManagement #TechUpdate #arint_info
-
AIエージェントに組み込んだAgent SkillsをSkillOpsで評価する
https://qiita.com/licux/items/9c640601a1dde7c18d9f?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items -
ИИ пишет код, а кто инженерит ИИ?
Как отличить инжениринг AI-систем от применения ИИ в разработке и почему coding agents в 2026 году требуют больше контроля, тестов и архитектурных границ
https://habr.com/ru/articles/1045090/
#SE_for_AI #AI_for_SE #агентная_программная_инженерия #coding_agents #MLOps #LLMOps #AI_governance #управление_рисками_ИИ
-
Не надо встраивать ИИ в каждую корпоративную систему, это архитектурная ошибка
Главная мысль этой статьи простая: не надо встраивать ИИ в каждую корпоративную систему как отдельный самостоятельный AI-контур. Пользователь должен видеть ИИ там, где работает: в CRM, СЭД, ITSM, мониторинге, портале или корпоративном чате. Но модели, GPU, gateway, лимиты, аудит, политики доступа, RAG, журналы и ответственность должны жить в отдельном корпоративном слое. ИИ должен быть доступен из каждой корпоративной системы, но не должен принадлежать ни одной из них. Как бухгалтер работает и в 1С, и в почте, и на корпоративном портале, так и корпоративный ИИ должен уметь работать с разными системами, не превращаясь в модуль одной из них. Это статья про архитектурную ошибку, которую мы почти совершили в Monq, про зоопарк корпоративных копилотов, теневой ИИ, GPU-инфраструктуру, AI Gateway, безопасность, аудит и новый слой между людьми и информационными системами.
https://habr.com/ru/companies/monq/articles/1038986/
#LLM #платформа #llmops #ai #aiагенты #open_source #opensource #monq #unica #ии
-
Self-Hosted Observability for Large Language Models Emerges
Developers can now watch their LLM work locally using self-hosted Langfuse. This gives more control over data and insights. Learn how it works.
#LLMOps, #Langfuse, #vLLM, #OpenSource, #AI
https://newsletter.tf/self-hosted-llm-observability-langfuse-vllm/
-
Running your Large Language Models locally with Langfuse is now easier. This new method gives developers more control over their data compared to cloud services.
#LLMOps, #Langfuse, #vLLM, #OpenSource, #AI
https://newsletter.tf/self-hosted-llm-observability-langfuse-vllm/ -
Shipping an LLM agent without knowing if it breaks on edge cases is a gamble. LangWatch lets you simulate thousands of conversations, version prompts, and catch regressions before they hit production. Open-source, self-hostable, and built for real agentic pipelines. What's in the stack? Next tweet. #LLMOps
-
AI systems are no longer just models.
Models can propose.
Systems must verify.#AI #AISecurity #AIGovernance #AgenticAI #RuntimeVerification #AISecurity #CyberSecurity #SoftwareEngineering #MachineLearning #MLOps #LLMOps #RiskManagement
-
AI systems are no longer just models.
Models can propose.
Systems must verify.#AI #AISecurity #AIGovernance #AgenticAI #RuntimeVerification #AISecurity #CyberSecurity #SoftwareEngineering #MachineLearning #MLOps #LLMOps #RiskManagement
-
A #CodePen-style #live #IDE for #building, #testing, and #debugging Model Context Protocol (#MCP) #servers — #LLM #agnostic — https://github.com/AnandPilania/mcp-live-playground
#modelcontextprotocol #sandbox #webide #llmagnostic #llmops #aitools #apiinterface #ai #BuildInPublic #nodejs #javascript
-
A #CodePen-style #live #IDE for #building, #testing, and #debugging Model Context Protocol (#MCP) #servers — #LLM #agnostic — https://github.com/AnandPilania/mcp-live-playground
#modelcontextprotocol #sandbox #webide #llmagnostic #llmops #aitools #apiinterface #ai #BuildInPublic #nodejs #javascript
