I am trying to use datasets.load_dataset to load multiple big files from disk. I noticed it could merge the content of the files together to be a single dataset. And then I called the datasets.map to tokenize the dataset. My question is: If the total content of the files are much bigger than RAM, …

How to handle big data?

lhoestq May 25, 2023, 3:08pm 7

Topic		Replies	Views
Dataset.map hangs on tokenization (relatively small dataset) 🤗Datasets	2	2044	April 22, 2022
Big text dataset loading for training 🤗Datasets	2	280	May 7, 2025
Support of very large dataset? 🤗Datasets	12	10535	August 24, 2022
Datasets map tokenization throws OSError: No space left on device 🤗Datasets	8	2581	March 19, 2025
Best practices for a large dataset 🤗Datasets	7	3188	May 6, 2025