Roadmap
We've started small-scale pretraining experiments to settle the data mixture, and we'll grow our ablations as we head toward a full-scale run.
- Data mixture and scheduling ablationsIn progress
- Hyperparameter tuning and scaling experiments
- Pretraining scaling experiments
- Final pretraining run
- Post-training
Experiment log
First results coming soon
We'll post write-ups, configs and training curves here as our data ablations finish.