open-thoughts

> [!NOTE] > We have released a paper for OpenThoughts! See our paper here. This model is a fine-tuned version of Qwen/Qwen2.5-7B-Instruct on the OpenThoughts-114k dataset dataset. The dataset is derived by distilling DeepSeek-R1 using the data pipeline available on github. More info about the dataset can be found on the dataset card at OpenThoughts-114k dataset. This model improves upon the Bespoke-Stratos-7B model, which used 17k examples (Bespoke-Stratos-17k dataset). The numbers reported in the table below are evaluated with our open-source tool Evalchemy. | | AIME24 | MATH500 | GPQA-Diamond | LCBv2 Easy | LCBv2 Medium | LCBv2 Hard | LCBv2 All | | --------------------------- | -------- | ------- | ------------ | ----------- | ------------- | ----------- | ---------- | | OpenThinker-7B | 31.3 | 83.0 | 42.4 | 75.3 | 28.6 | 6.5 | 39.9 | | Bespoke-Stratos-7B | 22.7 | 79.6 | 38.9 | 71.4 | 25.2 | 0.8 | 35.8 | | DeepSeek-R1-Distill-Qwen-7B | 60 | 88.2 | 46.9 | 79.7 | 45.1 | 14.6 | 50.1 | | gpt-4o-0513 | 8.7 | 75.8 | 46.5 | 87.4 | 42.7 | 8.9 | 50.5 | | o1-mini | 64 | 85.6 | 60 | 92.8 | 74.7 | 39.8 | 72.8 | We are fully open-source. Our model weights, datasets, data generation code, evaluation code, and training code are all publicly available. | | Open Weights | Open Data | Open Code | |--|--------------|-----------| --------- | |OpenThinker-7B|✅|✅|✅ | |Bespoke-Stratos-7B|✅|✅|✅| |DeepSeek-R1-Distill-Qwen-7B|✅|❌|❌| |gpt-4o-0513|❌|❌|❌|❌| |o1-mini|❌|❌|❌|❌| We used four 8xH100 nodes to train the model for 20 hours. The following hyperparameters were used during training: - learningrate: 1e-05 - trainbatchsize: 1 - evalbatchsize: 8 - seed: 42 - distributedtype: multi-GPU - numdevices: 32 - gradientaccumulationsteps: 3 - totaltrainbatchsize: 96 - totalevalbatchsize: 256 - optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments - lrschedulertype: cosine - lrschedulerwarmupratio: 0.1 - numepochs: 3.0 - Transformers 4.46.1 - Pytorch 2.3.0 - Datasets 3.1.0 - Tokenizers 0.20.3 More info can be found in our repository: https://github.com/open-thoughts/open-thoughts. Links - 📝 OpenThoughts Paper - 📊 Open Thoughts Launch Blog Post - 💻 Open Thoughts GitHub Repository - 🧠 OpenThoughts-114k dataset - 🤖 OpenThinker-7B model - this model. - 📊 Bespoke-Stratos Blog Post - 🧠 Bespoke-Stratos-17k dataset - 🤖 Bespoke-Stratos-32B model - 🤖 Bespoke-Stratos-7B model