Offline preprocessing versus online data loading
Write a curation script, run offline preprocessing, and store a materialized training set. Then load data online for GPU training. Offline work includes annotation, filtering, mixing, tokenization, and synthetic data generation. Online work depends on the input, for example image decoding and tokenization. An arrow from training back to the curation script indicates a lot of iteration.
a lot of iteration...
Write
curation script
Offline
preprocessing
Materialized
training set
Online
data loading
Annotation, filtering, mixing,
tokenization, synthetic data
generation, ...
Depends on input data, e.g.,
image decode, tokenize, ...
Scroll across to follow the pipeline
→