#kserve — Public Fediverse posts

Red Hat and Tesla engineers tackled a real production problem together.

3x output tokens/sec, 2x faster TTFT on Llama 3.1 70B with KServe + llm-d + vLLM. Fixes pushed upstream to KServe along the way.

This is what open source looks like. 🤝 🚀

https://llm-d.ai/blog/production-grade-llm-inference-at-scale-kserve-llm-d-vllm

#RedHat #Tesla #RedHatAI #vLLM #Pytorch #Kubernetes #OpenShift #KServe #llmd #Llama #OpenSource

KServe joins CNCF as an incubating project

https://www.redhat.com/en/blog/kserve-joins-cncf-incubating-project

#RedHat #Kubernetes #OpenShift #OpenShiftAI #RedHatAI #CNCF #KServe #Inference #ModelServing

🙌 Huge thanks to everyone who contributed to this journey from writing code, reviewing docs, to supporting governance and community growth.

Stay tuned! We’ll be publishing a detailed announcement blog soon with more insights on what this means for users, contributors, and the future of model serving on Kubernetes.

For now: thank you to the community for making this possible. 💙

#KServe #CNCF #OpenSource #ModelServing #AI #MLOps #CloudNative #Kubeflow #Kubernetes #k8s Kubeflow

A huge thank you to Kevin Wang and Faseela K from the CNCF TOC for all the hard work. It’s been such a pleasure collaborating with you both on this milestone. Thank you to all the community members who have contributed!

This is a big step for the KServe community, and we’re excited about the road ahead in making cloud-native model serving more accessible and production-ready for everyone.

#KServe #CNCF #OpenSource #ModelServing #AI #MLOps #CloudNative CNCF Kubernetes Kubeflow

Big thanks to everyone contributing code, reviews, and ideas — this integration is shaping up to be a game-changer for 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀-𝗻𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝘀𝗲𝗿𝘃𝗶𝗻𝗴. Stay tuned for next release!

#KServe #llmd #GenerativeAI #MLOps #Kubernetes #ModelServing #AIInfrastructure