#sglang — Public Fediverse posts on home.social

Habr @[email protected] · 2026-05-21 · 07:02 UTC

Архитектура AI-сервисов: почему монолит убивает latency и GPU

Ваш AI‑чат или автокомплит тормозит при 50 запросах в секунду? Монолит убивает GPU и латенси? В этом туториале — реальная архитектура low‑latency инференса на high‑load: почему изолированный inference‑bundle вместо монолита, как выбрать между vLLM и SGLang без маркетинга, зачем нужны continuous batching и admission control. Читать разбор

https://habr.com/ru/companies/otus/articles/1031286/

#AIсервисы #LLM #инференс #highload #latency #GPU #vLLM #SGLang #continuous_batching #admission_control

#admission_control #continuous_batching #sglang #vllm #gpu #latency

Analyst207 @[email protected] · 2026-04-20 · 18:57 UTC

SGLang Flaw Enables Remote Code Execution via Malicious Model Files

A single malicious file can become a powerful gateway for attackers to run arbitrary commands on vulnerable machines - and a newly disclosed flaw in SGLang, CVE-2026-5760, reveals just how easily this can happen through specially crafted GGUF model files. This highly severe vulnerability, scoring 9.8 out of 10.0, enables remote code…

https://osintsights.com/sglang-flaw-enables-remote-code-execution-via-malicious-model-files?utm_source=mastodon&utm_medium=social

#RemoteCodeExecution #Cve20265760 #CommandInjection #Gguf #Sglang

#remotecodeexecution #cve20265760 #commandinjection #gguf #sglang

Analyst207 @[email protected] · 2026-04-20 · 18:57 UTC

SGLang Flaw Enables Remote Code Execution via Malicious Model Files

A single malicious file can become a powerful gateway for attackers to run arbitrary commands on vulnerable machines - and a newly disclosed flaw in SGLang, CVE-2026-5760, reveals just how easily this can happen through specially crafted GGUF model files. This highly severe vulnerability, scoring 9.8 out of 10.0, enables remote code…

https://osintsights.com/sglang-flaw-enables-remote-code-execution-via-malicious-model-files?utm_source=mastodon&utm_medium=social

#RemoteCodeExecution #Cve20265760 #CommandInjection #Gguf #Sglang

#remotecodeexecution #cve20265760 #commandinjection #gguf #sglang

Rost Glukhov @[email protected] · 2026-03-24 · 08:03 UTC

Install SGLang with uv, pip, or Docker; configure YAML and server flags; then serve Hugging Face LLMs with an OpenAI-compatible API plus native /generate and offline Engine examples.

#Cheatsheet #Self-Hosting #LLM #AI #AI Coding #DevOps #Docker #sglang #openai #SelfHosting

https://www.glukhov.org/llm-hosting/sglang/

#cheatsheet #self #llm #ai #devops #docker

Rost Glukhov @[email protected] · 2026-03-24 · 08:03 UTC

Install SGLang with uv, pip, or Docker; configure YAML and server flags; then serve Hugging Face LLMs with an OpenAI-compatible API plus native /generate and offline Engine examples.

#Cheatsheet #Self-Hosting #LLM #AI #AI Coding #DevOps #Docker #sglang #openai #SelfHosting

https://www.glukhov.org/llm-hosting/sglang/

#cheatsheet #self #llm #ai #devops #docker

Rost Glukhov @[email protected] · 2026-03-24 · 08:03 UTC

Install SGLang with uv, pip, or Docker; configure YAML and server flags; then serve Hugging Face LLMs with an OpenAI-compatible API plus native /generate and offline Engine examples.

#Cheatsheet #Self-Hosting #LLM #AI #AI Coding #DevOps #Docker #sglang #openai #SelfHosting

https://www.glukhov.org/llm-hosting/sglang/

#cheatsheet #self #llm #ai #devops #docker

Rost Glukhov @[email protected] · 2026-03-24 · 08:03 UTC

Install SGLang with uv, pip, or Docker; configure YAML and server flags; then serve Hugging Face LLMs with an OpenAI-compatible API plus native /generate and offline Engine examples.

#Cheatsheet #Self-Hosting #LLM #AI #AI Coding #DevOps #Docker #sglang #openai #SelfHosting

https://www.glukhov.org/llm-hosting/sglang/

#selfhosting #openai #sglang #docker #devops #ai

Rost Glukhov @ros · 2026-03-24 · 08:03 UTC

Install SGLang with uv, pip, or Docker; configure YAML and server flags; then serve Hugging Face LLMs with an OpenAI-compatible API plus native /generate and offline Engine examples.

#Cheatsheet #Self-Hosting #LLM #AI #AI Coding #DevOps #Docker #sglang #openai #SelfHosting

https://www.glukhov.org/llm-hosting/sglang/

#cheatsheet #self #llm #ai #devops #docker

GOSIM Foundation @[email protected] · 2026-03-17 · 11:02 UTC

SGLang and vLLM Workshops Coming to GOSIM Paris 2026！

The GOSIM Workshops have long been known for their diversity, hands-on learning, and interactivity, making them one of the most popular segments of the conference.

This May, the SGLang Workshop and vLLM Workshop will arrive at GOSIM Paris 2026, bringing together AI infrastructure developers from around the world to explore the latest advances in LLM inference systems.

Ticket purchase link:
https://eventbrite.com/e/gosim-paris-2026-tickets-1984013840806?aff=oddtdtcreator

#SGLang #vLLM

#sglang #vllm

GOSIM Foundation @[email protected] · 2026-03-17 · 11:02 UTC

SGLang and vLLM Workshops Coming to GOSIM Paris 2026！

The GOSIM Workshops have long been known for their diversity, hands-on learning, and interactivity, making them one of the most popular segments of the conference.

This May, the SGLang Workshop and vLLM Workshop will arrive at GOSIM Paris 2026, bringing together AI infrastructure developers from around the world to explore the latest advances in LLM inference systems.

Ticket purchase link:
https://eventbrite.com/e/gosim-paris-2026-tickets-1984013840806?aff=oddtdtcreator

#SGLang #vLLM

#sglang #vllm

GOSIM Foundation @[email protected] · 2026-03-17 · 11:02 UTC

SGLang and vLLM Workshops Coming to GOSIM Paris 2026！

The GOSIM Workshops have long been known for their diversity, hands-on learning, and interactivity, making them one of the most popular segments of the conference.

This May, the SGLang Workshop and vLLM Workshop will arrive at GOSIM Paris 2026, bringing together AI infrastructure developers from around the world to explore the latest advances in LLM inference systems.

Ticket purchase link:
https://eventbrite.com/e/gosim-paris-2026-tickets-1984013840806?aff=oddtdtcreator

#SGLang #vLLM

#sglang #vllm

GOSIM Foundation @[email protected] · 2026-03-12 · 10:32 UTC

🚀 Big news!
The SGLang Workshop & vLLM Workshop are coming to GOSIM Paris 2026! 🎉
🌐 A must-attend event for AI developers and open-source contributors worldwide
💡 Dive into cutting-edge topics: large model inference, agentic AI, and more
🎓 Hands-on sessions and discussions to bring high-value learning and networking

Get your early bird tickets now and enjoy the discount: https://eventbrite.com/e/gosim-paris-2026-tickets-1984013840806?aff=oddtdtcreator 🚀

#GOSIMParis2026 #SGLang #vLLM #AIWorkshop #OpenSourceAI

#gosimparis2026 #sglang #vllm #aiworkshop #opensourceai

GOSIM Foundation @[email protected] · 2026-03-12 · 10:32 UTC

🚀 Big news!
The SGLang Workshop & vLLM Workshop are coming to GOSIM Paris 2026! 🎉
🌐 A must-attend event for AI developers and open-source contributors worldwide
💡 Dive into cutting-edge topics: large model inference, agentic AI, and more
🎓 Hands-on sessions and discussions to bring high-value learning and networking

Get your early bird tickets now and enjoy the discount: https://eventbrite.com/e/gosim-paris-2026-tickets-1984013840806?aff=oddtdtcreator 🚀

#GOSIMParis2026 #SGLang #vLLM #AIWorkshop #OpenSourceAI

#gosimparis2026 #sglang #vllm #aiworkshop #opensourceai

GOSIM Foundation @[email protected] · 2026-03-12 · 10:32 UTC

🚀 Big news!
The SGLang Workshop & vLLM Workshop are coming to GOSIM Paris 2026! 🎉
🌐 A must-attend event for AI developers and open-source contributors worldwide
💡 Dive into cutting-edge topics: large model inference, agentic AI, and more
🎓 Hands-on sessions and discussions to bring high-value learning and networking

Get your early bird tickets now and enjoy the discount: https://eventbrite.com/e/gosim-paris-2026-tickets-1984013840806?aff=oddtdtcreator 🚀

#GOSIMParis2026 #SGLang #vLLM #AIWorkshop #OpenSourceAI

#gosimparis2026 #sglang #vllm #aiworkshop #opensourceai

Reddit Tech VN Bot @[email protected] · 2026-01-29 · 17:38 UTC

Kimi K2.5 chạy trên ktkernel + sglang đạt 16 token/giây nhưng thiếu thẻ mở <think> trong chuỗi phản hồi, chỉ có </think>. Điều này gây lỗi với Open WebUI, Cline do parser không nhận diện được phần reasoning. Dù đã dùng --reasoning-parser kimi_k2 nhưng không hiệu quả. Cần tìm cách khôi phục thẻ mở <think>. #KimiK25 #SGLang #LocalLLM #AIInference #reasoning #hỗ_trợ_AI #mô_hình_ngôn_ngữ #trí_tuệ_nhân_tạo #thiếu_thẻ_thinking

https://www.reddit.com/r/LocalLLaMA/comments/1qqebfh/kim

#kimik25 #sglang #localllm #aiinference #reasoning #hỗ_trợ_ai

Reddit Tech VN Bot @[email protected] · 2026-01-23 · 11:15 UTC

DFlash: Hệ thống giải mã suy đoán theo kiểu khuếch tán, tạo block token cùng lúc thay vì từng token. Dùng draft model nhẹ để tạo block, kiểm nghiệm bằng LLM đích – tăng độ chấp nhận và hiệu suất, đặc biệt với văn cảnh dài & batch lớn. Hỗ trợ Qwen3-4B/8B/30B, tích hợp với SGLang, hỗ trợ streaming và sinh code dài. Hiệu quả cao trong sinh code và đầu ra cấu trúc. Code, checkpoint đã công bố, hướng dẫn huấn luyện sắp ra mắt. #DFlash #LLM #SpeculativeDecoding #Qwen3 #SGLang #AI #MachineLearning #Trí

#dflash #llm #speculativedecoding #qwen3 #sglang #ai

Reddit Tech VN Bot @[email protected] · 2026-01-23 · 10:18 UTC

DFlash: Hệ thống giải mã giả định theo phong cách khuếch tán, tạo khối token cùng lúc thay vì từng token. Dựa trên Qwen3 (4B, 8B, Coder-30B) và tích hợp với SGLang, cho tốc độ nhanh hơn, độ chấp nhận cao hơn – lý tưởng cho sinh mã và đầu ra cấu trúc. Hỗ trợ streaming, batch lớn. Mã nguồn đã mở, hướng dẫn train sắp ra mắt. #DFlash #LLM #AI #SpeculativeDecoding #Qwen3 #SGLang #TríTuệNhânTạo #MôHìnhNgônNgữ #GiảiMãKhối #KhuếchTán

https://www.reddit.com/r/LocalLLaMA/comments/1qkmzqv/dflash_diffusion

#dflash #llm #ai #speculativedecoding #qwen3 #sglang

Reddit Tech VN Bot @[email protected] · 2026-01-19 · 14:15 UTC

Intel phát hành bản cập nhật LLM-Scaler-Omni, nâng cao hiệu suất cho ComfyUI & SGLang trên GPU Arc.

#Intel #AI #LLM #GPU #ComfyUI #SGLang #CôngNghệ #OpenSource

https://www.reddit.com/r/LocalLLaMA/comments/1qh3oj0/intel_llmscaleromni_update_brings_comfyui_sglang/

#intel #ai #llm #gpu #comfyui #sglang

Reddit Tech VN Bot @[email protected] · 2025-12-29 · 08:16 UTC

So sánh thời gian khởi động torch.compile: SGLang vs vLLM

Người dùng báo cáo sự khác biệt lớn khi chạy Gemma 3 12B:
- vLLM (mặc định compile): ~1 phút.
- SGLang (không compile): ~1 phút 30 giây.
- SGLang (có compile, bs 1-16): ~6 phút.

Dù SGLang cho hiệu suất tăng 5-15% ở batch size thấp, chi phí khởi động lại quá cao. Nguyên nhân có thể do vLLM dùng "piecewise compilation" nhanh hơn, trong khi SGLang gắn chặt compile với CUDA graph toàn phần.

#AI #MachineLearning #SGLang #vLLM #TorchCompile

#ai #machinelearning #sglang #vllm #torchcompile

Reddit Tech VN Bot @[email protected] · 2025-12-24 · 23:22 UTC

Người dùng gặp lỗi khi chạy mô hình NVFP4 trên 2× RTX Pro 6000 Blackwell (96GB) với SGLang. Cần cưỡng bức quantization `modelopt_fp4` để khởi động thành công. Cảm báo DeepGemm & FP8-KV có thể ảnh hưởng độ chính xác. Cả 2 GPU hoạt động 100% ngay cả khi rỗi. VRAM sử dụng ~87–88GB. #GPU #AI #Blackwell #NVFP4 #SGLang #VấnNạnAI

https://www.reddit.com/r/LocalLLaMA/comments/1puzsm5/2_rtx_pro_6000_blackwell_96gb_sglang_nvfp4_loads/

#gpu #ai #blackwell #nvfp4 #sglang #vấnnạnai

Reddit Tech VN Bot @[email protected] · 2025-12-21 · 01:16 UTC

🔥 MiMo-V2-Flash & SGLang đạt tốc độ ấn tượng trên 4x 6000 Blackwell:
- **4K token**: 100.2 t/s | **8K**: 88.2 | **16K**: 67.0 | **64K**: 31.7 | **100K**: 24.5
- Độ chính xác ≈ 2.2–2.5.
🚀 Công nghệ Triton Attention tối ưu hiệu suất cho mô hình ngôn ngữ lớn.

#AI #LLM #SGLang #ViDiT #CôngNghệ #MáyHọc #TânCôngNghệ

https://www.reddit.com/r/LocalLLaMA/comments/1prt5qz/mimov2flash_sglang_mtp_triton_attention/

#ai #llm #sglang #vidit #congnghệ #mayhọc

AI Daily Post @[email protected] · 2025-11-30 · 05:00 UTC

Indian developers report that AI‑driven assistants let them spend less time typing and more time learning. From SGLang to Tensoic AI, interns say generative tools boost productivity and skill growth. Curious how the balance is shifting? Read the full story. #AIDrivenAssistants #GenerativeTools #SGLang #TensoicAI

🔗 https://aidailypost.com/news/indian-developers-say-ai-helps-them-learn-more-they-code-less

#aidrivenassistants #generativetools #sglang #tensoicai

AI Daily Post @[email protected] · 2025-11-30 · 05:00 UTC

Indian developers report that AI‑driven assistants let them spend less time typing and more time learning. From SGLang to Tensoic AI, interns say generative tools boost productivity and skill growth. Curious how the balance is shifting? Read the full story. #AIDrivenAssistants #GenerativeTools #SGLang #TensoicAI

🔗 https://aidailypost.com/news/indian-developers-say-ai-helps-them-learn-more-they-code-less

#aidrivenassistants #generativetools #sglang #tensoicai

Reddit Tech VN Bot @[email protected] · 2025-11-26 · 20:18 UTC

SGLang vừa giải quyết ổn định FP8 cho huấn luyện RL, phát hiện vấn đề nằm ở bước lượng tử hóa (quantization step). Đây là bước tiến lớn cho RLHF và tinh chỉnh RL cục bộ, giúp đơn giản hóa việc sử dụng độ chính xác hỗn hợp.
#SGLang #FP8 #RLTraining #Quantization #AI #MachineLearning #HuấnLuyệnRL #TríTuệNhânTạo #HọcMáy

https://www.reddit.com/r/LocalLLaMA/comments/1p7h5ah/sglang_just_solved_fp8_stability_for_rl_training/

#sglang #fp8 #rltraining #quantization #ai #machinelearning

Reddit Tech VN Bot @[email protected] · 2025-11-08 · 19:16 UTC

Có giải pháp vLLM/sgLang cho vấn đề gọi công cụ hallucinations không? Kimi K2 Thinking đang gặp vấn đề với gọi công cụ, làm cho nó không phù hợp với các ứng dụng như Roo Code / Aider. #vLLM #sgLang #KimiK2 #Hallucinations #CôngNghệ #TríTuệNhânTạo #AI #MachineLearning

https://www.reddit.com/r/LocalLLaMA/comments/1orwcc8/kimi_k2_thinking_is_there_currently_a_vllmsglang/

#vllm #sglang #kimik2 #hallucinations #congnghệ #trituệnhantạo

Reddit Tech VN Bot @[email protected] · 2025-11-08 · 18:23 UTC

💬🤖 Cộng đồng hỏi: “Kimi K2 Thinking” có chạy được với vLLM hoặc sglang, hỗ trợ gọi tool (tool‑calling) mà không bị hoang tưởng chưa? Hình như vấn đề mọc từ cách gọi tool và sai lệch trong grammar. Kimi hiện đang cố gắng áp dụng quy tắc grammar để sửa lỗi, còn nguồn tài nguyên hạn chế khi không hỗ trợ gọi tool. #KimiK2 #vLLM #sgLang #toolcalling #AI #NhómAI #ToolsLlama #AIhub #TechNews 🚀

https://www.reddit.com/r/LocalLLaMA/comments/1orwcc8/kimi_k2_thinking_is_there_currently_a_vllmsglang/

#kimik2 #vllm #sglang #toolcalling #ai #nhomai

Reddit Tech VN Bot @[email protected] · 2025-10-10 · 06:17 UTC

Olla v0.0.19 ra mắt, đã tích hợp hỗ trợ native SGLang và Lemonade. Phiên bản này tiếp tục là proxy LLM nhanh, cho phép gắn kết Ollama, LM Studio, LiteLLM, vLLM và OpenWebUI qua API tương thích OpenAI. Bạn có thể đổi mô hình, khởi động lại không cần, chỉ cần cập nhật cấu hình trong Olla. Hãy thử và cho phản hồi! #Olla #SGLang #Lemonade #LLM #OpenWebUI #AI #CôngNghệ #TríTuệNhânTạo

https://www.reddit.com/r/LocalLLaMA/comments/1o2t72w/olla_v0019_is_out_with_sglang_lemonade_support/

#olla #sglang #lemonade #llm #openwebui #ai