indonlp

Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages Cendol is an open-source collection of fine-tuned generative large language models in Indonesian languages covering decoder-only and encoder-decoder transformer model architectures ranging in scale from 300 million to 13 billion parameters. This is the repository for the 1.2B Cendol mT5-large Instruct model. Links to other models can be found below. Model Details Note: Use of Cendol is licensed under the Apache 2.0 license IndoNLP developed and publicly released the Cendol family of large language models (LLMs), a collection of pretrained and fine-tuned generative text models ranging in scale from 560 million to 13 billion parameters. Cendol models cover two instruction-tuned versions: 1. Cendol-Instruct that is instruction-tuned on tasks-specific NLP data such as sentiment analysis, topic modeling, machine translation, summarization, question answering, paraphrasing, etc 2. Cendol-Chat that is continuously instruction-tuned from Cendol-Instruct on general knowledge and human-centric prompts. Both Cendol-Instruct and Cendol-Chat are designed for a single-turn conversation. Cendol outperforms open-source multilingual and region-specific LLMs on most benchmarks we tested by a huge margin, with the smaller version ( -4 | |Cendol mT5-base Instruct|Cendol Collection v1|580M|Fully-Finetuned|3.0 x 10 -4 | |Cendol mT5-large Instruct|Cendol Collection v1|1.2B|Fully-Finetuned|3.0 x 10 -4 | |Cendol mT5-xl Instruct|Cendol Collection v1|3.7B|Fully-Finetuned|3.0 x 10 -4 | |Cendol mT5-xxl Instruct|Cendol Collection v1|13B|LoRA|2.0 x 10 -4 | |Cendol LLaMA-2 (7B) Instruct|Cendol Collection v1|7B|Fully-Finetuned|2.0 x 10 -5 | |Cendol LLaMA-2 (7B) Indonesian-Vocab Instruct|Cendol Collection v1|7B|Fully-Finetuned|2.0 x 10 -5 | |Cendol LLaMA-2 (13B) Instruct|Cendol Collection v1|13B|LoRA|2.0 x 10 -5 | |Cendol mT5-small Chat|Cendol Collection v2|300M|Fully-Finetuned|3.0 x 10 -5 | |Cendol mT5-base Chat|Cendol Collection v2|580M|Fully-Finetuned|3.0 x 10 -5 | |Cendol mT5-large Chat|Cendol Collection v2|1.2B|Fully-Finetuned|3.0 x 10 -5 | |Cendol mT5-xl Chat|Cendol Collection v2|3.7B|Fully-Finetuned|3.0 x 10 -5 | |Cendol mT5-xxl Chat|Cendol Collection v2|13B|LoRA|2.0 x 10 -4 | |Cendol LLaMA-2 (7B) Chat|Cendol Collection v2|7B|Fully-Finetuned|1.0 x 10 -5 | |Cendol LLaMA-2 (13B) Chat|Cendol Collection v2|13B|LoRA|2.0 x 10 -4 | Model Dates Cendol was trained between October 2023 and January 2024. License Use of Cendol is licensed under the Apache 2.0 license Research Paper "Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages" Intended Use Intended Use Cases Cendol is intended for research use especially on Indonesian languages. Cendol models are intended for a single turn instruction, with Cendol-Instruct models can be used for task-specific instruction, while Cendol-Chat models can be used for general knowledge instruction. Out-of-scope Uses Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English and Indonesian languages. Use in any other way that is prohibited by the Acceptable Use Policy and Licensing Agreement for Cendol. In this section, we report the results for the Cendol models on large-scale NLU and NLG benchmarks. For all the evaluations, we use our internal evaluations library. Ethical Considerations and Limitations Cendol is a new technology that carries risks with its use. Testing conducted to date has been in Indonesian, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Cendol’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Cendol, developers should perform safety testing and tuning tailored to their specific applications of the model. Citation If you are using any resources including Cendol models, code, or data, please cite the following articles: Additionally, if you are inspired by our work on region-specific language models especially for Indonesian and its local languages, please also consider citing the following articles: