llmcompressor.transformers.data.razorcal
Classes:
-
RazorCalDataset–Child text generation class for the RazorCal calibration corpus
RazorCalDataset
Bases: TextGenerationDataset
Child text generation class for the RazorCal calibration corpus
RazorCal is the multi-domain calibration corpus released with the RAZOR expert pruning method (https://github.com/nick7nlp/RAZOR). It contains 2,048 chat samples across seven domains: 512 Coding samples and 256 each of Math, Science (STEM), Chinese-STEM, Instruction Following, Tool Calling and World Knowledge. Samples are rendered with the processor's chat template, forwarding any tool definitions recorded alongside the sample; samples which fail to render fall back to their raw message contents.
The corpus is published
(https://huggingface.co/datasets/inference-optimization/RazorCal) with one
row per sample, keeping the domain and sub_source attribution columns.
Tool call arguments and tool definition parameters are recorded as JSON
strings and are decoded back into objects before rendering. Only the
train split exists; use num_calibration_samples to control how many
of the samples are used for calibration.
Parameters:
-
dataset_args(DatasetArguments) –configuration settings for dataset loading
-
split(str) –split from dataset to load, for instance
trainortrain[:512]. Only thetrainsplit is available -
processor(Processor) –processor or tokenizer to use on dataset