vllm.model_executor.layers.quantization.dequant_linear
¶
Linear layers that a checkpoint stores in FP8 and the model runs in BF16.
Classes:
-
DequantLinearMethod–A layer with an FP8 checkpoint weight that runs in BF16.
Functions:
-
bf16_linear–build(quant_config=...), a linear layer that runs in BF16: with the
DequantLinearMethod
¶
Bases: LinearMethodBase
A layer with an FP8 checkpoint weight that runs in BF16.
base, the checkpoint's linear method, created the checkpoint's
parameters, so weight and its scale load independently (in any order
and any number of load_weights calls). The weight may also arrive whole
in BF16, as RL trainers send it; it then needs no scale. Post-load
processing dequantizes an FP8 weight (per-tensor, per-channel, block or MX
scales) to BF16 and retires the scales. A reload first restores the
checkpoint layout (direct reload views the BF16 storage as FP8, layerwise
reload restages the layer), so each update dequantizes what it loaded.
Methods:
-
wrap–Run a built layer with an FP8 checkpoint weight in BF16. Returns
Source code in vllm/model_executor/layers/quantization/dequant_linear.py
wrap(layer)
classmethod
¶
Run a built layer with an FP8 checkpoint weight in BF16. Returns whether the layer runs this way.
Source code in vllm/model_executor/layers/quantization/dequant_linear.py
_dequantize(layer)
¶
The FP8 weight times its scale, in the layer's dtype. A block scale
covers weight_block_size (the last blocks may be partial); a 1-D scale
covers a channel or a merged shard (per-tensor).
Source code in vllm/model_executor/layers/quantization/dequant_linear.py
_load_weight_in_sent_dtype(layer)
¶
Keep a whole weight in the dtype it is sent in: FP8, or BF16 (no scale
needed), on storage sized for BF16. The loader is bound to the layer, so
layerwise reload's metadata does not keep the layer alive
(sanitize_layer_refs).
Source code in vllm/model_executor/layers/quantization/dequant_linear.py
_view_as(param, dtype)
¶
param's storage viewed in dtype with its shape, if it fits.
Source code in vllm/model_executor/layers/quantization/dequant_linear.py
_weight_loader_in_sent_dtype(layer, param, loaded_weight, *args, **kwargs)
¶
View the parameter in the sent dtype (on its storage when it fits).
Source code in vllm/model_executor/layers/quantization/dequant_linear.py
bf16_linear(build, quant_config)
¶
build(quant_config=...), a linear layer that runs in BF16: with the
checkpoint's FP8 weight (DequantLinearMethod.wrap), else unquantized.
Any other quantized layer (e.g. online quantization) is rebuilt
unquantized.