#k8s — Public Fediverse posts

Learn how to monitor LLM inference in production using Prometheus and Grafana. Track p95 latency, tokens/sec, queue duration, and KV cache usage across vLLM, TGI, and llama.cpp. Includes PromQL examples, dashboards, alerts, Docker & Kubernetes setups.

#Monitoring #Hosting #Self-Hosting #LLM #AI #DevOps #Docker #K8S #Prometheus #Grafana #observability #kubernetes #vllm

https://www.glukhov.org/observability/monitoring-llm-inference-prometheus-grafana/

See. This is some 4D chess right here. #amutable's post on what you can do with #varlink and systemd-report... Full system API, fully extensible, declarative, and without ssh! 🤯 Makes controller pattern of #k8s look like a toddler's toy. 🤣

https://amutable.com/blog/amutable-foundations-technical-overview#:~:text=To%20address,system,-%2E