saltlux

Introduction We introduce luxia-21.4b-alignment-v1.0, an instruction-tuned and alignment model based on luxia-21.4b. Please refer to the evaluation results table for details. Instruction Fine-tuning Strategy We utilize state-of-the-art instruction fine-tuning methods including supervised fine-tuning (SFT) and direct preference optimization (DPO) Data Contamination Test Results Results will be updated soon. License - saltlux/luxia-21.4b-alignment-v1.0: apache-2.0 Contact Us ### Any questions and suggestions are welcomed at the discussion tab.

Introduction We introduce LUXIA-21.4B-Alignment, a large language model (LLM) with 21.4 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. It's demonstrates unparalleled state-of-the-art performance in models with parameters under 35B, and it also outperformed the 72B model and the 34Bx2 MoE (Mixture of Experts) model. Please refer to the evaluation results table for details. The luxia-21.4b-alignment model is derived from the luxia-21.4b-instruct model through DPO training, and the luxia-21.4b-instruct model is an SFT trained version of the luxia-21.4b model. We plan to release both the pretrained model and the instruction-tuned model soon. We created the base model by expanding the layers through a passthrough method based on the internlm2-20b-llama model. And to recover the performance of the created model, we conducted continual pretraining. luxia-21.4b-instruct model We utilize state-of-the-art instruction fine-tuning methods including supervised fine-tuning (SFT). We used a mixture of the following datasets - c-s-ale/alpaca-gpt4-data - Open-Orca/SlimOrca - in-house generated data utilizing Metamath luxia-21.4b-alignment model We utilize state-of-the-art instruction fine-tuning methods including direct preference optimization (DPO). We used a mixture of the following datasets - jondurbin/truthy-dpo-v0.1 - abacusai/ARCDPOFewShot - abacusai/HellaSwagDPOFewShot Data Contamination Test Results We generate our contamination numbers using https://github.com/swj0419/detect-pretrain-code-contamination/tree/master, with internlm2-20b-llama as our reference model. luxia-21.4b-alignment-v1.2 has the following results: | Model | ARC | MMLU | TruthfulQA | GSM8K | |--------------------------------------|-------|---------|------------|--------| | luxia-21.4b-alignment-v1.2 | 0.00 | 0.07 | 0.13 | 0.34 | Open LLM Leaderboard Evaluation Results | Model | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K | |--------------------------------------|-------|-----------|---------|------------|------------|--------| | luxia-21.4b-alignment-v1.2 | 77.73 | 90.86 | 67.86 | 79.16 | 86.27 | 66.94 | License - saltlux/luxia-21.4b-alignment-v1.2: apache-2.0 Contact Us ### Any questions and suggestions are welcomed at the discussion tab.