Vllm - DEV Community

Skip to content

DEV Community

👋 Sign in for the ability to sort posts by relevant, latest, or top.

EME GUG

Oct 26

Your Local LLM Isn't Bad at Tool Calling, Your Pipeline Is: Fixing Tool-Call Accuracy in vLLM

#llm #vllm #python #ai

7 min read

Runcrate

Oct 26

vLLM on three GPUs: Qwen2.5-7B tokens per second and cost per 1M tokens on RTX A6000, L40S and H100

#vllm #llm #gpu #benchmark

5 min read

Oct 24

Two vLLM servers on one GPU: the startup check that bites, and the fix

#vllm #llm #gpu #mlops

2 min read

jamilxt

Oct 22

llm-d vs Plain vLLM: A Practical Kubernetes Inference Quickstart

#llmd #vllm #kubernetes #ai

8 min read

Oct 21

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

#ai #machinelearning #vllm #llm

14 min read

Oct 20

Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

#ai #machinelearning #vllm #performance

12 min read

EME GUG

Oct 20

Why LLM Inference Engines Are Fast: Continuous Batching, PagedAttention and Prefix Caching Explained with vLLM

#llm #vllm #inference #performance

7 min read

Oct 18

A 3090 vs an L40S: 1.09x at batch 1, 0.46x at batch 128

#llm #gpu #vllm #benchmark

2 min read

orca forge

Oct 19

Why reasoning_effort behaves differently across providers

#llm #openai #vllm #api

5 min read

munimv2

Oct 13

vLLM's --api-key leaves /invocations, /tokenize and /pause open. Lock it down with an allowlist proxy

#vllm #llm #security #abotwrotethis

3 min read

Oct 12

Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost

#modeldistillation #quantization #vllm #awq

10 min read

Suresh Kumar Pallapothu

Oct 16

Day 16: Local Model Execution — Running Open-Weight LLMs Securely with Ollama and vLLM

#ai #ollama #vllm #30dayaiseries

4 min read

munimv2

Oct 11

vLLM 0.30 and older: 30+ new advisories, most with no CVE ID, and your scanner can't see them

#vllm #security #selfhosted #abotwrotethis

4 min read

aivrar

Oct 9

Running vLLM natively on Windows

#vllm #windows #cuda #llm

6 min read

Rigorous 14 GiB TPU v5e memory budget constraints

xbill for Google Developer Experts

Oct 13

Small Apache-2.0 Models Against Gemma 4 E2B on One TPU v5e: Which Ones Keep Up?

#tpu #vllm #gemma #llm

10 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.