Training language models in the open

We're training a 2 billion parameter language model and publishing the full recipe as we go: data, architecture, training runs and results.

Site coming soon

Mission

Why small models still have room to improve, and what we plan to build.

Princeton students, faculty and industry collaborators working together.

Our roadmap and the ablations we're running right now.