ZeusLabs

Qwen 2.5 72B base model, trained for two epochs on the Chronos Divergence dataset using ChatML. It works well for roleplaying and storywriting as well as general assistant tasks. This model uses `ChatML` - below is an example. It is a preset in many frontends. Quantizations Please note that we tested this model with a 5.0bpw EXL2 quant. Results are not expected to be the same when going below this quanitzation. Thanks to our model quanters! Sampling Settings Here are some settings that work well with this model: Higher temp gives more uniqueness and less repetition. Please do not take these settings as the "best" - your system prompt matters significantly, and if you're roleplaying use the Basic system prompt in SillyTavern. You can also try other samplers like Top P. Note that Presence Penalty works with Repetition Penalty Range. Credit Thank you to my team consisting of @ToastyPigeon, @Fizzarolli, and myself @elinas. Additional thanks to @AlpinDale and the rest of the PygmalionAI team for graciously providing the compute to finetune this model! Thank you to anthracite-org as well for sponsoring this model. We used a combination of provided logs and WizardLM evol both cleaned up and de-slopped. Thanks to Anthropic and OpenAI for the models used to generate synthetic and partially synthetic data to train this model. Thanks Elon Musk for being based enough to train AI that compares to the top models. If you have any questions or concerns, please post in the community tab. DISCLAIMER: Outputs generated by the model are not reflective of our views.

The original model, LLaMA 1 was pre-trained at a sequence length of 2048 tokens. We went through two individual runs, targeting a sequence length of 16,384 which is a significant increase over the original length. While it was originally pre-trained on 1.4T tokens, it was shown to respond positively to our 500M token train and will coherently write and keep the same writing format (granted some caveats) up to 12K tokens relatively consistently. Chronos-Divergence-33B is a one of a kind model which is based on the original Chronos-33B and now focuses on prompt adherence for roleplay and storywriting. It was trained at 16,834 tokens and can go up to around 12,000 tokens before any deterioration without the use of RoPE or other model extending techniques. The unique aspect of this model is that is has little to no "GPT-isms" or commonly referred to "slop" which are repetitive phrases many modern LLMs output due to their pre-training and finetuning datasets. We completely cleaned our datasets and relied on the original "charm" of the L1 series and might bring this to more of the smaller models if this gains traction. It also avoids "purple prose" in the same way. RoPE or RULER has not been tested as we are satisfied with our results, we will also run evaluations, but are not expecting much from a dated model, focused on RP intelligence. Next steps would be to implement GQA (Grouped Query Attention) to as the number of tokens you input increases, so will memory usage, and this technique has been shown to reduce memory burden. This will require significant effort on our part (help welcome!) and we hope that quantizations will be sufficient in the meantime. The datasets used do not have a planned release date, though it is less the data and more the technique that was able to make this "dated" model very special and unlike many of us have experienced before due to the modernization added to the model without the common phrases GPTs like to output today, though making it uncensored as a result. Without spoiling anything, the name of the model and presented character have meaning... Look up Steins;Gate if you are not familiar :) This model uses `ChatML` - below is an example. It is a preset in many frontends. Quantization Please note that we tested this model in BF16/FP16 and 8bit. Results are not expected to be the same when going below this quanitzation. Sampling Settings Here are some settings that work well with this model: Credit Thank you to my team consisting of @Fizzarolli and @ToastyPigeon and myself @elinas. Fizz graciously provided compute for us to run this (dumb), but fun experiment on, while Toasty assisted in dataset preperation! I ran the MLOps in the meantime. Please be mindful of the license. This is strictly non-commercial by Meta LLaMA terms, but free to use at your own leisure personally. If you have any questions or concerns, please post in the community tab. DISCLAIMER: Outputs generated by the model are not reflective of our views.