Skip to content

llmcompressor.transformers.data.perfectblend

Classes:

PerfectBlendDataset 

PerfectBlendDataset(
    dataset_args: DatasetArguments,
    split: str,
    processor: Processor,
)

Bases: TextGenerationDataset

Child text generation class for the open-perfectblend dataset

Parameters:

  • dataset_args (DatasetArguments) –

    configuration settings for dataset loading

  • split (str) –

    split from dataset to load, for instance test or train[:5%]

  • processor (Processor) –

    processor or tokenizer to use on dataset

Source code in src/llmcompressor/transformers/data/perfectblend.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
def __init__(
    self, dataset_args: "DatasetArguments", split: str, processor: Processor
):
    dataset_args = deepcopy(dataset_args)
    dataset_args.dataset = "mlabonne/open-perfectblend"
    dataset_args.text_column = "conversations"

    super().__init__(dataset_args=dataset_args, split=split, processor=processor)

    if (
        self.tokenizer is not None
        and getattr(self.tokenizer, "chat_template", None) is None
    ):
        self.tokenizer.chat_template = self.DEFAULT_CHAT_TEMPLATE
        logger.warning(
            "tokenizer.chat_template is not set, using default chat template for "
            f"{self.__class__.__name__}"
        )