datasets.Dataset
The class Dataset supplied by Hugging Face streamlines dataset transformations and is convertible to a torch.utils.data.DataLoader or a tf.data.dataset.
def test_method(sample):
sample["test"] = [sample["text"], "blabla"]
return sample
dataset = Dataset.from_dict({"text": ["Hello world", "Hugging Face is great"]})
dataset = dataset.map(test_method, batched=True)
The code will generate **ArrowInvalid**: cannot mix list and non-list, non-null values. Without batched=True, the list is attached to each item.