Tomi Kokkonen

Handling massive datasets can be time-consuming and error-prone. We are introducing multiple additions to Valohai designed to streamline ML workflows involving a massive number of files, from dataset creation and preprocessing to model training and data lineage tracking.

Collecting, cleaning and labeling data is one of the most time-consuming problems in data science and this is especially true in NLP. Recently, we've seen data scientists utilize large language models such as OpenAI's GPT-4 to help produce datasets to train smaller NLP models that solve a more specific task.