All Repos
LLM FrameworksPythonApache-2.0

vllm

vllm-project/vllm · Updated

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project/vllm is a Python repository in the LLM Frameworks category with 95,842 stars and 24,127 forks on GitHub, released under the Apache-2.0 license, last updated today.

#amd#blackwell#cuda#deepseek#deepseek-v3#gpt#gpt-oss#inference
View on GitHub 96kGitHub stars

How to install vllm in your AI client

Setup for Claude Code, Claude Desktop, Cursor, Codex, Windsurf, Cline and VS Code — copy-paste ready.

How to install vllm in Claude CodeAnthropic's agentic CLI

  1. Clone & open

    bash
    git clone https://github.com/vllm-project/vllm.git
cd vllm
claude

  2. Let Claude read it

    Ask “/init” to generate a CLAUDE.md, then have Claude explain, run, or extend the codebase. It reads files on demand.

Documentation

Full README

Easy, fast, and cheap LLM serving for everyone

| <b>Documentation</b> | <b>Blog</b> | <b>Paper</b> | <b>Twitter/X</b> | <b>User Forum</b> | <b>Developer Slack</b> |

🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more. For events, please visit vllm.ai/events to join us.

About

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.

vLLM is fast with:

  • State-of-the-art serving throughput
  • Efficient management of attention key and value memory with **PagedAttention**
  • Continuous batching of incoming requests, chunked prefill, prefix caching
  • Fast and flexible model execution with piecewise and full CUDA/HIP graphs
  • Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more
  • Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
  • Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
  • Speculative decoding including n-gram, suffix, EAGLE, DFlash
  • Automatic kernel generation and graph-level transformations using torch.compile
  • Disaggregated prefill, decode, and encode

vLLM is flexible and easy to use with:

  • Seamless integration with popular Hugging Face models
  • High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
  • Tensor, pipeline, data, expert, and context parallelism for distributed inference
  • Streaming outputs
  • Generation of structured outputs using xgrammar or guidance
  • Tool calling and reasoning parsers
  • OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
  • Efficient multi-LoRA support for dense and MoE layers
  • Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.

vLLM seamlessly supports 200+ model architectures on Hugging Face, including:

  • Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
  • Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
  • Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
  • Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
  • Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
  • Reward and classification models (e.g., Qwen-Math)

Find the full list of supported models here.

More LLM Frameworks repositories

Catalog data refreshed . Something wrong? Report it.