Skip to content

vLLM

compressed_tensors_moe_w4a8_fp8

Initializing search

Home
User Guide
Developer Guide
Benchmarking
API Reference
CLI Reference
Community

vLLM

Home
User Guide
User Guide
- Getting Started
  Getting Started
  - Quickstart
  - Installation
    
    Installation
    
    GPU
    
    CPU
    
    TPU
  - Examples
    
    Examples
    
    Applications
    Applications
    
    Chatbot
    
    Rag
    
    Basic
    Basic
    
    Offline Inference
    
    Online Serving
    
    Deployment
    Deployment
    
    Async LLM Streaming
    
    Helm Charts
    
    LLM Engine Example
    
    Sagemaker-Entrypoint
    
    Disaggregated
    Disaggregated
    
    Disaggregated Encoder
    
    Disaggregated Prefill
    
    Disaggregated Serving
    
    Ec Both Encoder
    
    Disaggregated Prefill V1
    
    Flexkv Connector
    
    KV Load Failure Recovery Test
    
    LMCache Examples
    
    Mooncake Connector
    
    P2P NCCL Xpyd
    
    Features
    Features
    
    Automatic Prefix Caching
    
    Batch Invariance
    
    Context Extension
    
    Data Parallel
    
    Kv Events
    
    Logging Configuration
    
    Custom Logits Processors
    
    LoRA
    
    Offline Inference with the OpenAI Batch file format
    
    Pause Resume
    
    Profiling
    
    Prompt Embed
    
    Reset Kv
    
    Sharded State
    
    Speculative Decoding
    
    Structured Outputs
    
    Tensorize vLLM Model
    
    Torchrun
    
    Generate
    Generate
    
    Batched Chat Completions Online
    
    Multimodal
    
    Qwen 1M Offline
    
    Token Generation Client
    
    Observability
    Observability
    
    Monitoring Dashboards
    
    Metrics
    
    Setup OpenTelemetry POC
    
    Prometheus and Grafana
    
    Pooling
    Pooling
    
    Classify
    
    Embed
    
    Plugin
    
    Reward
    
    Score
    
    Token Classify
    
    Token Embed
    
    Ray Serving
    Ray Serving
    
    Batch LLM Inference
    
    Elastic Ep
    
    Multi-Node-Serving
    
    Ray Serve Deepseek
    
    Run Cluster
    
    Reasoning
    Reasoning
    
    OpenAI Chat Completion Tool Calls With Reasoning
    
    OpenAI Chat Completion With Reasoning
    
    OpenAI Chat Completion With Reasoning Streaming
    
    OpenAI Responses Client
    
    RL
    RL
    
    RLHF Async New APIs
    
    RLHF Http IPC
    
    RLHF Http NCCL
    
    RLHF IPC
    
    RLHF NCCL
    
    RLHF NCCL Fsdp Ep
    
    Routed Experts E2E
    
    Skip Loading Weights In Engine Init
    
    Speech To Text
    Speech To Text
    
    Lid
    
    OpenAI
    
    Realtime
    
    Tool Calling
    Tool Calling
    
    Chat With Tools Offline
    
    OpenAI Chat Completion Client With Tools
    
    OpenAI Chat Completion Client With Tools Required
    
    OpenAI Chat Completion Client With Tools Xlam
    
    OpenAI Chat Completion Client With Tools Xlam Streaming
    
    OpenAI Responses Client With Mcp Tools
    
    OpenAI Responses Client With Tools
- General
  General
- Inference and Serving
  Inference and Serving
  - Offline Inference
  - OpenAI-Compatible Server
  - Context Parallel Deployment
  - Data Parallel Deployment
  - Troubleshooting distributed deployments
  - Expert Parallel Deployment
  - Parallelism and Scaling
  - Integrations
    Integrations
    
    Claude Code
    
    Codex
    
    LangChain
    
    LlamaIndex
- Deployment
  Deployment
  - Using Docker
  - Using Kubernetes
  - Using Nginx
  - Frameworks
    Frameworks
    
    Anyscale
    
    AnythingLLM
    
    AutoGen
    
    BentoML
    
    Cerebrium
    
    Chatbox
    
    Dify
    
    dstack
    
    Haystack
    
    Helm
    
    Hugging Face Inference Endpoints
    
    LiteLLM
    
    Lobe Chat
    
    LWS
    
    Modal
    
    Open WebUI
    
    Retrieval-Augmented Generation
    
    RunPod
    
    SkyPilot
    
    Streamlit
    
    NVIDIA Triton
  - Integrations
    Integrations
    
    AIBrix
    
    NVIDIA Dynamo
    
    KAITO
    
    KServe
    
    Kthena
    
    KubeAI
    
    KubeRay
    
    Llama Stack
    
    llm-d
    
    llmaz
    
    Production stack
- Training
  Training
- Configuration
  Configuration
- Models
  Models
  - Supported Models
  - Generative Models
  - Pooling Models
    
    Pooling Models
    
    Classification Usages
    
    Embedding Usages
    
    Reward Usages
    
    Scoring Usages
    
    Specific Model Examples
    
    Token Classification Usages
    
    Token Embedding Usages
  - Extensions
    Extensions
    
    Loading model weights with fastsafetensors
    
    Loading Model Weights with InstantTensor
    
    Loading models with Run:ai Model Streamer
    
    Loading models with CoreWeave's Tensorizer
  - Hardware Supported Models
    Hardware Supported Models
    
    CPU - Intel® Xeon®
    
    XPU - Intel® GPUs
    
    TPU
- Features
  Features
Developer Guide
Developer Guide
- General
  General
- Model Implementation
  Model Implementation
- CI
  CI
- Design Documents
  Design Documents
Benchmarking
Benchmarking
API Reference
API Reference
- vllm
  vllm
  - beam_search
  - collect_env
  - connections
  - env_override
  - envs
  - exceptions
  - forward_context
  - logger
  - logits_process
  - logprobs
  - model_inspection
  - outputs
  - pooling_params
  - sampling_params
  - scalar_type
  - scripts
  - sequence
  - tasks
  - version
  - assets
    
    assets
    
    audio
    
    base
    
    image
    
    video
  - benchmarks
    
    benchmarks
    
    latency
    
    mm_processor
    
    plot
    
    serve
    
    startup
    
    throughput
    
    datasets
    
    datasets
    
    create_txt_slices_dataset
    
    datasets
    
    utils
    
    lib
    
    lib
    
    endpoint_request_func
    
    ready_checker
    
    utils
    
    sweep
    
    sweep
    
    cli
    
    param_sweep
    
    plot
    
    plot_pareto
    
    serve
    
    serve_workload
    
    server
    
    startup
    
    utils
  - compilation
    
    compilation
    
    backends
    
    base_static_graph
    
    caching
    
    codegen
    
    compiler_interface
    
    counter
    
    cuda_graph
    
    decorators
    
    monitor
    
    partition_rules
    
    piecewise_backend
    
    wrapper
    
    passes
    
    passes
    
    fx_utils
    
    inductor_pass
    
    pass_manager
    
    vllm_inductor_pass
    
    fusion
    
    fusion
    
    act_quant_fusion
    
    allreduce_rms_fusion
    
    attn_quant_fusion
    
    collective_fusion
    
    matcher_utils
    
    minimax_qk_norm_fusion
    
    mla_attn_quant_fusion
    
    qk_norm_rope_fusion
    
    rms_quant_fusion
    
    rocm_aiter_fusion
    
    rope_kvcache_fusion
    
    sequence_parallelism
    
    ir
    
    ir
    
    clone_elimination
    
    inplace_functionalization
    
    lowering_pass
    
    utils
    
    utility
    
    utility
    
    fix_functionalization
    
    noop_elimination
    
    post_cleanup
    
    scatter_split_replace
    
    split_coalescing
  - config
    
    config
    
    attention
    
    cache
    
    compilation
    
    device
    
    ec_transfer
    
    kernel
    
    kv_events
    
    kv_transfer
    
    load
    
    lora
    
    mamba
    
    model
    
    model_arch
    
    multimodal
    
    observability
    
    offload
    
    parallel
    
    pooler
    
    profiler
    
    quantization
    
    reasoning
    
    scheduler
    
    speculative
    
    speech_to_text
    
    structured_outputs
    
    utils
    
    vllm
    
    weight_transfer
  - device_allocator
    
    device_allocator
    
    cumem
  - distributed
    
    distributed
    
    communication_op
    
    kv_events
    
    nixl_utils
    
    parallel_state
    
    stateless_coordinator
    
    utils
    
    device_communicators
    
    device_communicators
    
    all2all
    
    all_reduce_utils
    
    base_device_communicator
    
    cpu_communicator
    
    cuda_communicator
    
    cuda_wrapper
    
    custom_all_reduce
    
    flashinfer_all_reduce
    
    mnnvl_compat
    
    pynccl
    
    pynccl_allocator
    
    pynccl_wrapper
    
    quick_all_reduce
    
    ray_communicator
    
    shm_broadcast
    
    shm_object_storage
    
    symm_mem
    
    xpu_communicator
    
    ec_transfer
    
    ec_transfer
    
    ec_transfer_state
    
    ec_connector
    
    ec_connector
    
    base
    
    example_connector
    
    factory
    
    elastic_ep
    
    elastic_ep
    
    elastic_execute
    
    elastic_state
    
    standby_state
    
    eplb
    
    eplb
    
    async_worker
    
    eplb_communicator
    
    eplb_state
    
    eplb_utils
    
    rebalance_execute
    
    policy
    
    policy
    
    abstract
    
    default
    
    kv_transfer
    
    kv_transfer
    
    kv_transfer_state
    
    kv_connector
    
    kv_connector
    
    base
    
    factory
    
    utils
    
    v1
    
    v1
    
    base
    
    decode_bench_connector
    
    example_connector
    
    example_hidden_states_connector
    
    flexkv_connector
    
    lmcache_connector
    
    lmcache_mp_connector
    
    metrics
    
    multi_connector
    
    offloading_connector
    
    simple_cpu_offload_connector
    
    ssm_conv_transfer_utils
    
    hf3fs
    
    hf3fs
    
    hf3fs_client
    
    hf3fs_connector
    
    hf3fs_metadata_server
    
    utils
    
    utils
    
    common
    
    gather_scatter_helper
    
    hf3fs_mock_client
    
    lmcache_integration
    
    lmcache_integration
    
    multi_process_adapter
    
    utils
    
    vllm_v1_adapter
    
    mooncake
    
    mooncake
    
    mooncake_connector
    
    mooncake_utils
    
    stats
    
    store
    
    store
    
    connector
    
    data
    
    scheduler
    
    worker
    
    moriio
    
    moriio
    
    moriio_common
    
    moriio_connector
    
    moriio_engine
    
    nixl
    
    nixl
    
    connector
    
    metadata
    
    scheduler
    
    stats
    
    tp_mapping
    
    utils
    
    worker
    
    offloading
    
    offloading
    
    common
    
    metrics
    
    scheduler
    
    worker
    
    p2p
    
    p2p
    
    p2p_nccl_connector
    
    p2p_nccl_engine
    
    tensor_memory_pool
    
    weight_transfer
    
    weight_transfer
    
    base
    
    factory
    
    ipc_engine
    
    nccl_engine
    
    packed_tensor
  - engine
    
    engine
    
    arg_utils
    
    async_llm_engine
    
    llm_engine
    
    protocol
  - entrypoints
    
    entrypoints
    
    api_server
    
    chat_utils
    
    constants
    
    grpc_server
    
    launcher
    
    llm
    
    logger
    
    ssl
    
    utils
    
    anthropic
    
    anthropic
    
    api_router
    
    protocol
    
    serving
    
    cli
    
    cli
    
    collect_env
    
    launch
    
    main
    
    openai
    
    run_batch
    
    serve
    
    types
    
    benchmark
    
    benchmark
    
    base
    
    latency
    
    main
    
    mm_processor
    
    serve
    
    startup
    
    sweep
    
    throughput
    
    mcp
    
    mcp
    
    tool
    
    tool_server
    
    openai
    
    openai
    
    api_server
    
    cli_args
    
    fingerprint
    
    orca_metrics
    
    run_batch
    
    server_utils
    
    utils
    
    chat_completion
    
    chat_completion
    
    api_router
    
    batch_serving
    
    protocol
    
    serving
    
    stream_harmony
    
    completion
    
    completion
    
    api_router
    
    protocol
    
    serving
    
    engine
    
    engine
    
    protocol
    
    serving
    
    generate
    
    generate
    
    api_router
    
    factories
    
    generative_scoring
    
    generative_scoring
    
    api_router
    
    serving
    
    models
    
    models
    
    api_router
    
    protocol
    
    serving
    
    parser
    
    parser
    
    harmony_utils
    
    responses_parser
    
    realtime
    
    realtime
    
    api_router
    
    connection
    
    metrics
    
    protocol
    
    serving
    
    responses
    
    responses
    
    api_router
    
    context
    
    harmony
    
    protocol
    
    serving
    
    streaming_events
    
    utils
    
    speech_to_text
    
    speech_to_text
    
    api_router
    
    protocol
    
    serving
    
    speech_to_text
    
    pooling
    
    pooling
    
    factories
    
    typing
    
    utils
    
    base
    
    base
    
    io_processor
    
    protocol
    
    serving
    
    classify
    
    classify
    
    api_router
    
    io_processor
    
    protocol
    
    serving
    
    embed
    
    embed
    
    api_router
    
    io_processor
    
    protocol
    
    serving
    
    pooling
    
    pooling
    
    api_router
    
    io_processor
    
    protocol
    
    serving
    
    scoring
    
    scoring
    
    api_router
    
    io_processor
    
    protocol
    
    serving
    
    typing
    
    utils
    
    sagemaker
    
    sagemaker
    
    api_router
    
    serve
    
    serve
    
    cache
    
    cache
    
    api_router
    
    disagg
    
    disagg
    
    api_router
    
    mm_serde
    
    protocol
    
    serving
    
    elastic_ep
    
    elastic_ep
    
    api_router
    
    middleware
    
    instrumentator
    
    instrumentator
    
    basic
    
    health
    
    metrics
    
    offline_docs
    
    server_info
    
    lora
    
    lora
    
    api_router
    
    protocol
    
    profile
    
    profile
    
    api_router
    
    render
    
    render
    
    api_router
    
    serving
    
    rlhf
    
    rlhf
    
    api_router
    
    rpc
    
    rpc
    
    api_router
    
    sleep
    
    sleep
    
    api_router
    
    tokenize
    
    tokenize
    
    api_router
    
    protocol
    
    serving
  - inputs
    
    inputs
    
    engine
    
    llm
    
    preprocess
  - ir
    
    ir
    
    op
    
    tolerances
    
    util
    
    ops
    
    ops
    
    layernorm
  - kernels
    
    kernels
    
    aiter_ops
    
    oink_ops
    
    vllm_c
    
    xpu_ops
    
    helion
    
    helion
    
    case_key
    
    config_manager
    
    register
    
    utils
    
    ops
    
    ops
    
    silu_mul_fp8
    
    triton
    
    triton
    
    qkv_padded_fp8_quant
  - logging_utils
    
    logging_utils
    
    access_log_filter
    
    dump_input
    
    formatter
    
    lazy
    
    log_time
    
    torch_tensor
  - lora
    
    lora
    
    lora_model
    
    lora_weights
    
    model_manager
    
    peft_helper
    
    request
    
    resolver
    
    utils
    
    worker_manager
    
    layers
    
    layers
    
    base
    
    base_linear
    
    column_parallel_linear
    
    fused_moe
    
    logits_processor
    
    replicated_linear
    
    row_parallel_linear
    
    utils
    
    vocal_parallel_embedding
    
    ops
    
    ops
    
    torch_ops
    
    torch_ops
    
    lora_ops
    
    triton_ops
    
    triton_ops
    
    fp8_kernel_utils
    
    fused_moe_lora_fp8_op
    
    fused_moe_lora_op
    
    kernel_utils
    
    lora_expand_fp8_op
    
    lora_expand_op
    
    lora_kernel_metadata
    
    lora_shrink_fp8_op
    
    lora_shrink_op
    
    utils
    
    xpu_ops
    
    xpu_ops
    
    lora_ops
    
    punica_wrapper
    
    punica_wrapper
    
    punica_base
    
    punica_cpu
    
    punica_gpu
    
    punica_selector
    
    punica_xpu
    
    utils
  - model_executor
    
    model_executor
    
    custom_op
    
    parameter
    
    utils
    
    kernels
    
    kernels
    
    linear
    
    linear
    
    base
    
    mixed_precision
    
    mixed_precision
    
    allspark
    
    conch
    
    cpu
    
    cutlass
    
    dynamic_4bit
    
    exllama
    
    MPLinearKernel
    
    machete
    
    marlin
    
    triton_w4a16
    
    xpu
    
    mxfp8
    
    mxfp8
    
    emulation
    
    flashinfer
    
    Mxfp8LinearKernel
    
    marlin
    
    xpu
    
    nvfp4
    
    nvfp4
    
    base
    
    cutlass
    
    emulation
    
    fbgemm
    
    flashinfer
    
    marlin
    
    scaled_mm
    
    scaled_mm
    
    aiter
    
    BlockScaledMMLinearKernel
    
    cpu
    
    cutlass
    
    deep_gemm
    
    flashinfer
    
    marlin
    
    pytorch
    
    rocm
    
    ScaledMMLinearKernel
    
    triton
    
    xpu
    
    layers
    
    layers
    
    activation
    
    attention_layer_base
    
    batch_invariant
    
    conv
    
    deepseek_compressor
    
    deepseek_v4_attention
    
    kda
    
    layernorm
    
    lightning_attn
    
    linear
    
    logits_processor
    
    mhc
    
    mla
    
    resampler
    
    sparse_attn_indexer
    
    utils
    
    vocab_parallel_embedding
    
    attention
    
    attention
    
    attention
    
    chunked_local_attention
    
    cross_attention
    
    encoder_only_attention
    
    kv_transfer_utils
    
    mla_attention
    
    mm_encoder_attention
    
    static_sink_attention
    
    fla
    
    fla
    
    ops
    
    ops
    
    chunk
    
    chunk_delta_h
    
    chunk_o
    
    chunk_scaled_dot_kkt
    
    cumsum
    
    fused_gdn_prefill_post_conv
    
    fused_recurrent
    
    fused_sigmoid_gating
    
    index
    
    kda
    
    l2norm
    
    layernorm_guard
    
    op
    
    solve_tril
    
    utils
    
    wy_fast
    
    fused_moe
    
    fused_moe
    
    activation
    
    all2all_utils
    
    config
    
    cpu_fused_moe
    
    deep_gemm_utils
    
    fallback
    
    fused_batched_moe
    
    fused_humming_moe
    
    fused_moe
    
    fused_moe_method_base
    
    fused_moe_modular_method
    
    layer
    
    lora_context
    
    lora_experts_mixin
    
    modular_kernel
    
    moe_align_block_size
    
    moe_fused_mul_sum
    
    moe_permute_unpermute
    
    routed_experts_capturer
    
    topk_weight_and_reduce
    
    triton_cutlass_moe
    
    triton_deep_gemm_moe
    
    unquantized_fused_moe_method
    
    utils
    
    experts
    
    experts
    
    aiter_mxfp4_w4a8_moe
    
    batched_deep_gemm_moe
    
    cpu_moe
    
    cutlass_moe
    
    deep_gemm_moe
    
    flashinfer_cutedsl_batched_moe
    
    flashinfer_cutedsl_moe
    
    flashinfer_cutlass_moe
    
    gpt_oss_triton_kernels_moe
    
    marlin_moe
    
    nvfp4_emulation_moe
    
    ocp_mx_emulation_moe
    
    rocm_aiter_moe
    
    triton_moe
    
    trtllm_bf16_moe
    
    trtllm_fp8_moe
    
    trtllm_mxfp4_moe
    
    trtllm_nvfp4_moe
    
    xpu_moe
    
    oracle
    
    oracle
    
    fp8
    
    int8
    
    int_wna16
    
    mxfp4
    
    mxfp8
    
    nvfp4
    
    unquantized
    
    prepare_finalize
    
    prepare_finalize
    
    batched
    
    deepep_ht
    
    deepep_ll
    
    flashinfer_nvlink_one_sided
    
    flashinfer_nvlink_two_sided
    
    mori
    
    naive_dp_ep
    
    nixl_ep
    
    no_dp_ep
    
    router
    
    router
    
    aiter_shared_routed_fused_moe_router
    
    base_router
    
    custom_routing_router
    
    fused_moe_router
    
    fused_topk_bias_router
    
    fused_topk_router
    
    gate_linear
    
    grouped_topk_router
    
    router_factory
    
    routing_simulator_router
    
    zero_expert_router
    
    runner
    
    runner
    
    moe_runner
    
    moe_runner_interface
    
    shared_experts
    
    mamba
    
    mamba
    
    abstract
    
    gdn_linear_attn
    
    lamport_workspace
    
    linear_attn
    
    mamba_mixer
    
    mamba_mixer2
    
    mamba_utils
    
    short_conv
    
    ops
    
    ops
    
    causal_conv1d
    
    layernorm_gated
    
    mamba_ssm
    
    ssd_bmm
    
    ssd_chunk_scan
    
    ssd_chunk_state
    
    ssd_combined
    
    ssd_state_passing
    
    ssu_dispatch
    
    triton_helpers
    
    cpu
    
    cpu
    
    causal_conv1d
    
    gdn_attention
    
    recurrent_gated_delta_rule
    
    pooler
    
    pooler
    
    abstract
    
    activations
    
    common
    
    special
    
    seqwise
    
    seqwise
    
    heads
    
    methods
    
    poolers
    
    tokwise
    
    tokwise
    
    heads
    
    methods
    
    poolers
    
    quantization
    
    quantization
    
    awq
    
    awq_marlin
    
    awq_triton
    
    base_config
    
    bitsandbytes
    
    cpu_wna16
    
    experts_int8
    
    fbgemm_fp8
    
    fp8
    
    fp_quant
    
    gguf
    
    gptq
    
    gptq_marlin
    
    humming
    
    inc
    
    input_quant_fp8
    
    kv_cache
    
    modelopt
    
    moe_wna16
    
    mxfp4
    
    qutlass_utils
    
    schema
    
    torchao
    
    compressed_tensors
    
    compressed_tensors
    
    compressed_tensors
    
    triton_scaled_mm
    
    utils
    
    compressed_tensors_moe
    
    compressed_tensors_moe
    
    compressed_tensors_moe
    
    compressed_tensors_moe_w4a4_mxfp4
    
    compressed_tensors_moe_w4a4_nvfp4
    
    compressed_tensors_moe_w4a8_fp8
    
    compressed_tensors_moe_w4a8_int8
    
    compressed_tensors_moe_w8a8_fp8
    
    compressed_tensors_moe_w8a8_int8
    
    compressed_tensors_moe_w8a8_mxfp8
    
    compressed_tensors_moe_wna16
    
    compressed_tensors_moe_wna16_marlin
    
    schemes
    
    schemes
    
    compressed_tensors_24
    
    compressed_tensors_scheme
    
    compressed_tensors_w4a4_nvfp4
    
    compressed_tensors_w4a8_fp8
    
    compressed_tensors_w4a8_int
    
    compressed_tensors_w4a16_mxfp4
    
    compressed_tensors_w4a16_nvfp4
    
    compressed_tensors_w8a8_fp8
    
    compressed_tensors_w8a8_int8
    
    compressed_tensors_w8a8_mxfp8
    
    compressed_tensors_w8a16_fp8
    
    compressed_tensors_wNa16
    
    transform
    
    transform
    
    linear
    
    module
    
    utils
    
    schemes
    
    schemes
    
    linear_qutlass_nvfp4
    
    online
    
    online
    
    base
    
    fp8
    
    int8
    
    moe_base
    
    mxfp8
    
    quark
    
    quark
    
    quark
    
    quark_moe
    
    utils
    
    schemes
    
    schemes
    
    quark_ocp_mx
    
    quark_scheme
    
    quark_w4a8_mxfp4_fp8
    
    quark_w8a8_fp8
    
    quark_w8a8_int8
    
    turboquant
    
    turboquant
    
    centroids
    
    config
    
    quantizer
    
    utils
    
    utils
    
    allspark_utils
    
    flashinfer_fp4_moe
    
    flashinfer_mxint4_moe
    
    flashinfer_utils
    
    fp8_utils
    
    gptq_utils
    
    humming_utils
    
    int8_utils
    
    layer_utils
    
    machete_utils
    
    marlin_utils
    
    marlin_utils_fp4
    
    marlin_utils_fp8
    
    marlin_utils_test
    
    mxfp4_utils
    
    mxfp6_utils
    
    mxfp8_utils
    
    nvfp4_emulation_utils
    
    nvfp4_utils
    
    ocp_mx_utils
    
    quant_utils
    
    w8a8_utils
    
    rotary_embedding
    
    rotary_embedding
    
    base
    
    common
    
    deepseek_scaling_rope
    
    dual_chunk_rope
    
    dynamic_ntk_alpha_rope
    
    dynamic_ntk_scaling_rope
    
    ernie45_vl_rope
    
    fope
    
    gemma4_rope
    
    linear_scaling_rope
    
    llama3_rope
    
    llama4_vision_rope
    
    mrope
    
    mrope_interleaved
    
    ntk_scaling_rope
    
    phi3_long_rope_scaled_rope
    
    telechat3_scaling_rope
    
    xdrope
    
    yarn_scaling_rope
    
    model_loader
    
    model_loader
    
    base_loader
    
    bitsandbytes_loader
    
    default_loader
    
    dummy_loader
    
    ep_weight_filter
    
    gguf_loader
    
    runai_streamer_loader
    
    sharded_state_loader
    
    tensorizer
    
    tensorizer_loader
    
    utils
    
    weight_utils
    
    reload
    
    reload
    
    layerwise
    
    meta
    
    sanitize
    
    torchao_decorator
    
    types
    
    utils
    
    models
    
    models
    
    AXK1
    
    adapters
    
    afmoe
    
    aimv2
    
    apertus
    
    arcee
    
    arctic
    
    aria
    
    audioflamingo3
    
    aya_vision
    
    bagel
    
    baichuan
    
    bailing_moe
    
    bailing_moe_linear
    
    bamba
    
    bee
    
    bert
    
    bert_with_rope
    
    blip
    
    blip2
    
    bloom
    
    chameleon
    
    chatglm
    
    cheers
    
    clip
    
    cohere2_moe
    
    cohere2_vision
    
    cohere_asr
    
    cohere_eagle
    
    colbert
    
    colmodernvbert
    
    colpali
    
    colqwen3
    
    colqwen3_5
    
    commandr
    
    config
    
    conformer_encoder
    
    dbrx
    
    deepencoder
    
    deepencoder2
    
    deepseek_eagle
    
    deepseek_eagle3
    
    deepseek_mtp
    
    deepseek_ocr
    
    deepseek_ocr2
    
    deepseek_v2
    
    deepseek_v4
    
    deepseek_v4_mtp
    
    deepseek_vl2
    
    dots1
    
    dots_ocr
    
    eagle2_5_vl
    
    ernie
    
    ernie45
    
    ernie45_moe
    
    ernie45_vl
    
    ernie45_vl_moe
    
    ernie_mtp
    
    exaone
    
    exaone4
    
    exaone4_5
    
    exaone4_5_mtp
    
    exaone_moe
    
    exaone_moe_mtp
    
    extract_hidden_states
    
    fairseq2_llama
    
    falcon
    
    falcon_h1
    
    fireredasr2
    
    fireredlid
    
    flex_olmo
    
    funasr
    
    funaudiochat
    
    fuyu
    
    gemma
    
    gemma2
    
    gemma3
    
    gemma3_mm
    
    gemma3n
    
    gemma3n_audio_utils
    
    gemma3n_mm
    
    gemma4
    
    gemma4_mm
    
    gemma4_mtp
    
    glm
    
    glm4
    
    glm4_1v
    
    glm4_moe
    
    glm4_moe_lite
    
    glm4_moe_lite_mtp
    
    glm4_moe_mtp
    
    glm4v
    
    glm_ocr
    
    glm_ocr_mtp
    
    glmasr
    
    glmasr_utils
    
    gpt2
    
    gpt_bigcode
    
    gpt_j
    
    gpt_neox
    
    gpt_oss
    
    granite
    
    granite4_vision
    
    granite_speech
    
    granitemoe
    
    granitemoehybrid
    
    granitemoeshared
    
    gritlm
    
    grok1
    
    h2ovl
    
    hunyuan_v1
    
    hunyuan_vision
    
    hy_v3
    
    hy_v3_mtp
    
    hyperclovax
    
    hyperclovax_vision
    
    hyperclovax_vision_v2
    
    idefics2_vision_model
    
    idefics3
    
    interfaces
    
    interfaces_base
    
    intern_vit
    
    internlm2
    
    internlm2_ve
    
    interns1
    
    interns1_pro
    
    interns1_vit
    
    internvl
    
    iquest_loopcoder
    
    isaac
    
    jais
    
    jais2
    
    jamba
    
    jina
    
    jina_vl
    
    kanana_v
    
    keye
    
    keye_vl1_5
    
    kimi_audio
    
    kimi_k25
    
    kimi_k25_vit
    
    kimi_linear
    
    kimi_vl
    
    laguna
    
    lfm2
    
    lfm2_moe
    
    lfm2_siglip2
    
    lfm2_vl
    
    lightonocr
    
    llama
    
    llama4
    
    llama4_eagle
    
    llama_eagle
    
    llama_eagle3
    
    llava
    
    llava_next
    
    llava_next_video
    
    llava_onevision
    
    longcat_flash
    
    longcat_flash_mtp
    
    mamba
    
    mamba2
    
    medusa
    
    midashenglm
    
    mimo
    
    mimo_audio
    
    mimo_mtp
    
    mimo_v2
    
    mimo_v2_mtp
    
    mimo_v2_omni
    
    minicpm
    
    minicpm3
    
    minicpm_eagle
    
    minicpmo
    
    minicpmv
    
    minimax_m2
    
    minimax_text_01
    
    minimax_vl_01
    
    mistral
    
    mistral3
    
    mistral_eagle
    
    mistral_large_3
    
    mistral_large_3_eagle
    
    mixtral
    
    mllama4
    
    mlp_speculator
    
    modernbert
    
    module_mapping
    
    molmo
    
    molmo2
    
    moondream3
    
    moonvit
    
    mpt
    
    musicflamingo
    
    nano_nemotron_vl
    
    nemotron
    
    nemotron_h
    
    nemotron_h_mtp
    
    nemotron_nas
    
    nemotron_parse
    
    nemotron_vl
    
    nvlm_d
    
    olmo
    
    olmo2
    
    olmo_hybrid
    
    olmoe
    
    opencua
    
    openpangu
    
    openpangu_mtp
    
    openpangu_vl
    
    opt
    
    orion
    
    ouro
    
    ovis
    
    ovis2_5
    
    paddleocr_vl
    
    paligemma
    
    parakeet
    
    param2moe
    
    persimmon
    
    phi
    
    phi3
    
    phi3v
    
    phi4mm
    
    phi4mm_audio
    
    phi4mm_utils
    
    phi4siglip
    
    phimoe
    
    pixtral
    
    plamo2
    
    plamo3
    
    qianfan_ocr
    
    qwen
    
    qwen2
    
    qwen2_5_omni_thinker
    
    qwen2_5_vl
    
    qwen2_audio
    
    qwen2_moe
    
    qwen2_rm
    
    qwen2_vl
    
    qwen3
    
    qwen3_5
    
    qwen3_5_mtp
    
    qwen3_asr
    
    qwen3_asr_forced_aligner
    
    qwen3_asr_realtime
    
    qwen3_dflash
    
    qwen3_moe
    
    qwen3_next
    
    qwen3_next_mtp
    
    qwen3_omni_moe_thinker
    
    qwen3_vl
    
    qwen3_vl_moe
    
    qwen_vl
    
    radio
    
    registry
    
    rnj1
    
    roberta
    
    rvl
    
    sarvam
    
    seed_oss
    
    siglip
    
    siglip2navit
    
    skyworkr1v
    
    smolvlm
    
    solar
    
    stablelm
    
    starcoder2
    
    step1
    
    step3_text
    
    step3_vl
    
    step3p5
    
    step3p5_mtp
    
    step_vl
    
    tarsier
    
    telechat2
    
    teleflm
    
    terratorch
    
    ultravox
    
    utils
    
    vision
    
    voxtral
    
    voxtral_realtime
    
    voyage
    
    whisper
    
    whisper_causal
    
    whisper_utils
    
    zamba2
    
    transformers
    
    transformers
    
    base
    
    causal
    
    legacy
    
    moe
    
    multimodal
    
    pooling
    
    utils
    
    offloader
    
    offloader
    
    base
    
    prefetch
    
    prefetch_ops
    
    uva
    
    warmup
    
    warmup
    
    deep_gemm_warmup
    
    kernel_warmup
  - multimodal
    
    multimodal
    
    audio
    
    cache
    
    encoder_budget
    
    evs
    
    hasher
    
    image
    
    inputs
    
    parse
    
    registry
    
    utils
    
    video
    
    media
    
    media
    
    audio
    
    base
    
    connector
    
    image
    
    video
    
    processing
    
    processing
    
    context
    
    dummy_inputs
    
    inputs
    
    processor
  - parser
    
    parser
    
    abstract_parser
    
    minimax_m2_parser
    
    parser_manager
  - platforms
    
    platforms
    
    cpu
    
    cuda
    
    interface
    
    rocm
    
    tpu
    
    xpu
    
    zen_cpu
  - plugins
    
    plugins
    
    io_processors
    
    io_processors
    
    interface
    
    lora_resolvers
    
    lora_resolvers
    
    filesystem_resolver
    
    hf_hub_resolver
  - profiler
    
    profiler
    
    layerwise_profile
    
    utils
    
    wrapper
  - ray
    
    ray
    
    lazy_utils
    
    ray_env
  - reasoning
    
    reasoning
    
    abs_reasoning_parsers
    
    basic_parsers
    
    cohere_command_reasoning_parser
    
    deepseek_r1_reasoning_parser
    
    deepseek_v3_reasoning_parser
    
    ernie45_reasoning_parser
    
    gemma4_reasoning_parser
    
    gemma4_utils
    
    gptoss_reasoning_parser
    
    granite_reasoning_parser
    
    hunyuan_a13b_reasoning_parser
    
    hy_v3_reasoning_parser
    
    identity_reasoning_parser
    
    kimi_k2_reasoning_parser
    
    minimax_m2_reasoning_parser
    
    mistral_reasoning_parser
    
    nemotron_v3_reasoning_parser
    
    olmo3_reasoning_parser
    
    poolside_v1_reasoning_parser
    
    qwen3_reasoning_parser
    
    seedoss_reasoning_parser
    
    step3_reasoning_parser
    
    step3p5_reasoning_parser
  - renderers
    
    renderers
    
    base
    
    deepseek_v4
    
    deepseek_v32
    
    embed_utils
    
    grok2
    
    hf
    
    mistral
    
    params
    
    registry
    
    terratorch
    
    inputs
    
    inputs
    
    preprocess
    
    tokenize
  - tokenizers
    
    tokenizers
    
    deepseek_v4
    
    deepseek_v4_encoding
    
    deepseek_v32
    
    deepseek_v32_encoding
    
    detokenizer_utils
    
    fastokens
    
    grok2
    
    hf
    
    kimi_audio
    
    mistral
    
    protocol
    
    qwen_vl
    
    registry
  - tool_parsers
    
    tool_parsers
    
    abstract_tool_parser
    
    cohere_command_tool_parser
    
    deepseekv3_tool_parser
    
    deepseekv4_tool_parser
    
    deepseekv31_tool_parser
    
    deepseekv32_tool_parser
    
    ernie45_tool_parser
    
    functiongemma_tool_parser
    
    gemma4_tool_parser
    
    gemma4_utils
    
    gigachat3_tool_parser
    
    glm4_moe_tool_parser
    
    glm47_moe_tool_parser
    
    granite4_tool_parser
    
    granite_20b_fc_tool_parser
    
    granite_tool_parser
    
    hermes_tool_parser
    
    hunyuan_a13b_tool_parser
    
    hy_v3_tool_parser
    
    internlm2_tool_parser
    
    jamba_tool_parser
    
    kimi_k2_tool_parser
    
    lfm2_tool_parser
    
    llama4_pythonic_tool_parser
    
    llama_tool_parser
    
    longcat_tool_parser
    
    minimax_m2_tool_parser
    
    minimax_tool_parser
    
    mistral_tool_parser
    
    olmo3_tool_parser
    
    openai_tool_parser
    
    phi4mini_tool_parser
    
    poolside_v1_tool_parser
    
    pythonic_tool_parser
    
    qwen3coder_tool_parser
    
    qwen3xml_tool_parser
    
    seed_oss_tool_parser
    
    step3_tool_parser
    
    step3p5_tool_parser
    
    streaming
    
    structural_tag_registry
    
    utils
    
    xlam_tool_parser
  - tracing
    
    tracing
    
    otel
    
    utils
  - transformers_utils
    
    transformers_utils
    
    config
    
    config_parser_base
    
    dynamic_module
    
    gguf_utils
    
    model_arch_config_convertor
    
    processor
    
    repo_utils
    
    runai_utils
    
    s3_utils
    
    tokenizer
    
    utils
    
    chat_templates
    
    chat_templates
    
    registry
    
    configs
    
    configs
    
    AXK1
    
    afmoe
    
    arctic
    
    bagel
    
    chatglm
    
    cheers
    
    colmodernvbert
    
    colpali
    
    colqwen3
    
    deepseek_v4
    
    deepseek_vl2
    
    dotsocr
    
    eagle
    
    extract_hidden_states
    
    falcon
    
    fireredlid
    
    flex_olmo
    
    funaudiochat
    
    granite4_vision
    
    hunyuan_vl
    
    hy_v3
    
    hyperclovax
    
    isaac
    
    jais
    
    kimi_k25
    
    kimi_linear
    
    kimi_vl
    
    laguna
    
    lfm2_moe
    
    medusa
    
    midashenglm
    
    mimo_v2_omni
    
    mistral
    
    mlp_speculator
    
    moondream3
    
    moonvit
    
    nemotron
    
    nemotron_h
    
    olmo_hybrid
    
    ovis
    
    parakeet
    
    qianfan_ocr
    
    qwen3_5
    
    qwen3_5_moe
    
    qwen3_asr
    
    qwen3_next
    
    radio
    
    step3_vl
    
    step3p5
    
    tarsier2
    
    ultravox
    
    speculators
    
    speculators
    
    algos
    
    base
    
    processors
    
    processors
    
    bagel
    
    cheers
    
    cohere_asr
    
    deepseek_ocr
    
    deepseek_vl2
    
    fireredasr2
    
    fireredlid
    
    funasr
    
    glm4v
    
    granite4_vision
    
    h2ovl
    
    hunyuan_vl
    
    hunyuan_vl_image
    
    internvl
    
    isaac
    
    kimi_audio
    
    kimi_k25
    
    mimo_v2_omni
    
    moondream3
    
    nano_nemotron_vl
    
    nemotron_vl
    
    nvlm_d
    
    ovis
    
    ovis2_5
    
    pixtral
    
    qwen3_asr
    
    qwen_vl
    
    step3_vl
    
    voxtral
  - triton_utils
    
    triton_utils
    
    allocation
    
    importing
    
    jit_monitor
  - usage
    
    usage
    
    usage_lib
  - utils
    
    utils
    
    argparse_utils
    
    async_utils
    
    cache
    
    collection_utils
    
    counter
    
    cpu_resource_utils
    
    cpu_triton_utils
    
    deep_gemm
    
    flashinfer
    
    func_utils
    
    gc_utils
    
    hashing
    
    import_utils
    
    jsontree
    
    math_utils
    
    mem_constants
    
    mem_utils
    
    mistral
    
    multi_stream_utils
    
    nccl
    
    network_utils
    
    numa_utils
    
    nvtx_pytorch_hooks
    
    ompmultiprocessing
    
    platform_utils
    
    print_utils
    
    profiling
    
    registry
    
    serial_utils
    
    system_utils
    
    tensor_schema
    
    torch_utils
    
    tqdm_utils
  - v1
    
    v1
    
    cudagraph_dispatcher
    
    kv_cache_interface
    
    outputs
    
    request
    
    serial_utils
    
    utils
    
    attention
    
    attention
    
    backend
    
    selector
    
    backends
    
    backends
    
    cpu_attn
    
    fa_utils
    
    flash_attn
    
    flash_attn_diffkv
    
    flashinfer
    
    flex_attention
    
    gdn_attn
    
    linear_attn
    
    mamba1_attn
    
    mamba2_attn
    
    mamba_attn
    
    registry
    
    rocm_aiter_fa
    
    rocm_aiter_unified_attn
    
    rocm_attn
    
    short_conv_attn
    
    triton_attn
    
    turboquant_attn
    
    utils
    
    mla
    
    mla
    
    aiter_triton_mla
    
    compressor_utils
    
    cutlass_mla
    
    flashattn_mla
    
    flashinfer_mla
    
    flashinfer_mla_sparse
    
    flashmla
    
    flashmla_sparse
    
    indexer
    
    rocm_aiter_mla
    
    rocm_aiter_mla_sparse
    
    sparse_swa
    
    sparse_utils
    
    tokenspeed_mla
    
    triton_mla
    
    xpu_mla_sparse
    
    prefill
    
    prefill
    
    base
    
    flash_attn
    
    flashinfer
    
    registry
    
    selector
    
    tokenspeed_mla
    
    trtllm_ragged
    
    ops
    
    ops
    
    chunked_prefill_paged_decode
    
    common
    
    dcp_alltoall
    
    flashmla
    
    merge_attn_states
    
    paged_attn
    
    prefix_prefill
    
    rocm_aiter_mla_sparse
    
    triton_attention_helpers
    
    triton_decode_attention
    
    triton_merge_attn_states
    
    triton_prefill_attention
    
    triton_reshape_and_cache_flash
    
    triton_turboquant_decode
    
    triton_turboquant_store
    
    triton_unified_attention
    
    vit_attn_wrappers
    
    xpu_mla_sparse
    
    deepseek_v4_ops
    
    deepseek_v4_ops
    
    cache_utils
    
    fused_compress_quant_cache
    
    fused_indexer_q
    
    fused_indexer_q_cutedsl
    
    fused_inv_rope_fp8_quant
    
    fused_qk_rmsnorm
    
    core
    
    core
    
    block_pool
    
    encoder_cache_manager
    
    kv_cache_coordinator
    
    kv_cache_manager
    
    kv_cache_metrics
    
    kv_cache_utils
    
    single_type_kv_cache_manager
    
    sched
    
    sched
    
    async_scheduler
    
    interface
    
    output
    
    request_queue
    
    scheduler
    
    utils
    
    engine
    
    engine
    
    async_llm
    
    coordinator
    
    core
    
    core_client
    
    detokenizer
    
    exceptions
    
    input_processor
    
    llm_engine
    
    logprobs
    
    output_processor
    
    parallel_sampling
    
    tensor_ipc
    
    utils
    
    executor
    
    executor
    
    abstract
    
    multiproc_executor
    
    ray_env_utils
    
    ray_executor
    
    ray_executor_v2
    
    ray_utils
    
    uniproc_executor
    
    kv_offload
    
    kv_offload
    
    base
    
    factory
    
    reuse_manager
    
    cpu
    
    cpu
    
    common
    
    gpu_worker
    
    manager
    
    shared_offload_region
    
    spec
    
    policies
    
    policies
    
    arc
    
    base
    
    lru
    
    worker
    
    worker
    
    worker
    
    metrics
    
    metrics
    
    loggers
    
    perf
    
    prometheus
    
    ray_wrappers
    
    reader
    
    stats
    
    utils
    
    pool
    
    pool
    
    late_interaction
    
    metadata
    
    sample
    
    sample
    
    metadata
    
    rejection_sampler
    
    sampler
    
    thinking_budget_state
    
    logits_processor
    
    logits_processor
    
    builtin
    
    interface
    
    state
    
    ops
    
    ops
    
    bad_words
    
    logprobs
    
    penalties
    
    topk_topp_sampler
    
    topk_topp_triton
    
    simple_kv_offload
    
    simple_kv_offload
    
    copy_backend
    
    cuda_mem_ops
    
    manager
    
    metadata
    
    worker
    
    spec_decode
    
    spec_decode
    
    dflash
    
    draft_model
    
    eagle
    
    extract_hidden_states
    
    gemma4
    
    llm_base_proposer
    
    medusa
    
    metadata
    
    metrics
    
    ngram_proposer
    
    ngram_proposer_gpu
    
    suffix_decoding
    
    utils
    
    structured_output
    
    structured_output
    
    backend_guidance
    
    backend_lm_format_enforcer
    
    backend_outlines
    
    backend_types
    
    backend_xgrammar
    
    request
    
    utils
    
    worker
    
    worker
    
    block_table
    
    cp_utils
    
    cpu_model_runner
    
    cpu_worker
    
    dp_utils
    
    ec_connector_model_runner_mixin
    
    encoder_cudagraph
    
    encoder_cudagraph_defs
    
    gpu_input_batch
    
    gpu_model_runner
    
    gpu_ubatch_wrapper
    
    gpu_worker
    
    kv_connector_model_runner_mixin
    
    lora_model_runner_mixin
    
    mamba_utils
    
    tpu_input_batch
    
    ubatch_utils
    
    ubatching
    
    utils
    
    worker_base
    
    workspace
    
    xpu_model_runner
    
    xpu_worker
    
    gpu
    
    gpu
    
    async_utils
    
    attn_utils
    
    block_table
    
    buffer_utils
    
    cp_utils
    
    cudagraph_utils
    
    dp_utils
    
    eplb_utils
    
    input_batch
    
    kv_connector
    
    lora_utils
    
    model_runner
    
    pp_utils
    
    shutdown
    
    states
    
    structured_outputs
    
    warmup
    
    metrics
    
    metrics
    
    logits
    
    mm
    
    mm
    
    encoder_cache
    
    encoder_runner
    
    rope
    
    model_states
    
    model_states
    
    default
    
    interface
    
    mamba_hybrid
    
    whisper
    
    pool
    
    pool
    
    late_interaction_runner
    
    pooling_runner
    
    sample
    
    sample
    
    bad_words
    
    gumbel
    
    logit_bias
    
    logprob
    
    min_p
    
    output
    
    penalties
    
    prompt_logprob
    
    sampler
    
    states
    
    spec_decode
    
    spec_decode
    
    probabilistic_rejection_sampler_utils
    
    rejection_sampler
    
    synthetic_rejection_sampler_utils
    
    utils
    
    eagle
    
    eagle
    
    cudagraph
    
    eagle3_utils
    
    speculator
    
    utils
CLI Reference
CLI Reference
- vllm serve
- vllm chat
- vllm complete
- vllm run-batch
- vllm bench
  vllm bench
Community
Community
- Contact Us
- Meetups
- Sponsors
- Governance
  Governance
- Blog
- Forum
- Slack

Home
API Reference
vllm
model_executor
layers
quantization
compressed_tensors
compressed_tensors_moe

vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe.compressed_tensors_moe_w4a8_fp8 ¶

Made with Material for MkDocs