Skip to content

llmcompressor.transformers.data.wikitext

Classes:

  • WikiTextDataset –

    Child text generation class for the Open Platypus dataset

WikiTextDataset 

WikiTextDataset(
    dataset_args: DatasetArguments,
    split: str,
    processor: Processor,
)

Bases: TextGenerationDataset

Child text generation class for the Open Platypus dataset

Parameters:

  • dataset_args (DatasetArguments) –

    configuration settings for dataset loading

  • split (str) –

    split from dataset to load, for instance test or train[:5%]

  • processor (Processor) –

    processor or tokenizer to use on dataset

Source code in src/llmcompressor/transformers/data/wikitext.py 
21
22
23
24
25
26
27
28
29
30
31
32
def __init__(
    self, dataset_args: "DatasetArguments", split: str, processor: Processor
):
    dataset_args = deepcopy(dataset_args)
    dataset_args.dataset = "Salesforce/wikitext"
    dataset_args.text_column = "text"

    super().__init__(
        dataset_args=dataset_args,
        split=split,
        processor=processor,
    )