[Bug]: FP8 W8A8 quantized model doesn't work properly on V1 with default compilation_config (use_cudagraph = True)

[imkero]

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: TencentOS Server 3.1 (Final) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (TencentOS 8.5.0-18)
Clang version: 16.0.6 (Red Hat 16.0.6-2.module+el8.8.0+557+454507bd)
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.12.8 | packaged by conda-forge | (main, Dec  5 2024, 14:24:40) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.4.119-19.0009.40-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA L40

Nvidia driver version: 535.161.07
cuDNN version: Probably one of the following:
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              384
On-line CPU(s) list: 0-383
Thread(s) per core:  2
Core(s) per socket:  96
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
CPU family:          25
Model:               17
Model name:          AMD EPYC 9K84 96-Core Processor
Stepping:            0
CPU MHz:             2600.026
BogoMIPS:            5200.05
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            32768K
NUMA node0 CPU(s):   0-191
NUMA node1 CPU(s):   192-383
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid amd_dcm tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 avx512_bf16 clzero xsaveerptr wbnoinvd arat avx512vbmi umip avx512_vbmi2 vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.3
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.2.1                   pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torchaudio                2.5.1                    pypi_0    pypi
[conda] torchvision               0.20.1                   pypi_0    pypi
[conda] transformers              4.48.3                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.1.dev1+g0ccd876 (git sha: 0ccd876
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled

NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

• Using vLLM V1 with enforce_eager=False
• Model loaded and generate output without any error, but the output content is gibberish

An temporary approach?

Inspired by this comment, I turned compilation_config.use_cudagraph from True to False (diff: imkero@92116c3, should change the source code in vllm/config.py because it always override compilation_config), and then it works as expected.

vllm/vllm/config.py

Lines 3237 to 3249 in 009439c

	if envs.VLLM_USE_V1 and self.model_config is not None and \
	not self.model_config.enforce_eager:
	# NOTE(woosuk): Currently, we use inductor because the piecewise
	# CUDA graphs do not work properly with the custom CUDA kernels.
	# FIXME(woosuk): Disable inductor to reduce the compilation time
	# and avoid any potential issues with the inductor.
	self.compilation_config.custom_ops = ["none"]
	self.compilation_config.use_cudagraph = True
	self.compilation_config.use_inductor = True
	self.compilation_config.cudagraph_num_of_warmups = 1
	self.compilation_config.pass_config.enable_fusion = False
	self.compilation_config.pass_config.enable_reshape = False
	self.compilation_config.level = CompilationLevel.PIECEWISE

Should this problem be addressed by modifying compilation_config, or some bug should be fixed instead?

Code and model to reproduce

Model: nm-testing/DeepSeek-R1-Distill-Qwen-14B-FP8-Dynamic

Code:

vLLM latest main: 0ccd876, and this script:

import os
os.environ["VLLM_USE_V1"] = "1"
os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

MODEL = "nm-testing/DeepSeek-R1-Distill-Qwen-14B-FP8-Dynamic"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

def chat_template(question):
    return tokenizer.apply_chat_template(
        [
            {
                "role": "user",
                "content": question,
            },
        ],
        tokenize=False,
        add_generation_prompt=True,
    )

prompts = [
    chat_template(question)
    for question in [
        "Hello",
        "Which one is greater: 9.11 or 9.9?"
    ]
]
sampling_params = SamplingParams(temperature=0, max_tokens=128)

llm = LLM(
    model=MODEL,
    gpu_memory_utilization=0.9,
    max_model_len=1024,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Outputs

vLLM main branch code output:

Prompt: '<｜begin▁of▁sentence｜><｜User｜>Hello<｜Assistant｜>', Generated text: '开头\n空气质量格佯抽 bist exc� 3好00\n210的，的,100000<think>10<think>0100329003021 the0，111的1的1  2的1,1，，0的1 的,的1的的，ence，的 的0ence,\n1的4的的的1的,的,0201<think>，，1001，002的，的，的<think>的frac1，的<think><think>的的的的,'
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Which one is greater: 9.11 or 9.9?<｜Assistant｜>', Generated text: '开头\n reogra�001-001<think> record pre��使用2=1，00011000，，，11,,,0，,,，,1\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'

Exptected output:

Prompt: '<｜begin▁of▁sentence｜><｜User｜>Hello<｜Assistant｜>', Generated text: '<think>\n\n</think>\n\nHello! How can I assist you today? 😊'
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Which one is greater: 9.11 or 9.9?<｜Assistant｜>', Generated text: "<think>\nFirst, I'll compare the whole number parts of both numbers. Both 9.11 and 9.9 have the same whole number, which is 9.\n\nNext, I'll look at the decimal parts. In 9.11, the decimal part is 0.11, and in 9.9, it's 0.9.\n\nTo make a clear comparison, I'll express 0.9 as 0.90. Now, comparing 0.90 and 0.11, it's evident that 0.90 is greater than 0."

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tlrmchlsmth]

Thanks for the thorough bug report. I'm taking a look now. Have not been able to reproduce the issue so far -- output looks great using V1 on latest main on an H100

[tlrmchlsmth]

Update: I have a repro. It produces back output in the case that we fall back to torch._scaled_mm

[WoosukKwon]

Hmm... I wasn't able to reproduce the error on my local machine for some reason. My output looks correct:

Prompt: '<｜begin▁of▁sentence｜><｜User｜>Hello<｜Assistant｜>', Generated text: '<think>\n\n</think>\n\nHello! How can I assist you today? 😊'
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Which one is greater: 9.11 or 9.9?<｜Assistant｜>', Generated text: "<think>\nFirst, I'll compare the whole number parts of both numbers. Both 9.11 and 9.9 have the same whole number, which is 9.\n\nNext, I'll look at the decimal parts. In 9.11, the decimal part is 0.11, and in 9.9, it's 0.9.\n\nTo make the comparison easier, I'll express 0.9 as 0.90. Now, comparing 0.90 and 0.11, it's clear that 0.90 is greater than 0."

[tlrmchlsmth]

@WoosukKwon I think it should repro if you apply this patch. (Basically simulates being on an Ada Lovelace GPU on CUDA < 12.4)

diff --git a/vllm/_custom_ops.py b/vllm/_custom_ops.py
index e3e3c644f..56927f270 100644
--- a/vllm/_custom_ops.py
+++ b/vllm/_custom_ops.py
@@ -434,7 +434,7 @@ if hasattr(torch.ops._C, "ggml_dequantize"):

 # cutlass
 def cutlass_scaled_mm_supports_fp8(cuda_device_capability: int) -> bool:
-    return torch.ops._C.cutlass_scaled_mm_supports_fp8(cuda_device_capability)
+    retunr False


 def cutlass_scaled_mm_supports_block_fp8(cuda_device_capability: int) -> bool: