zehui127

Omni-DNA is a cross-modal, multi-task genomic foundation model designed to generalize across diverse genomic tasks. Unlike previous Genomic Foundation Models (GFMs), which require separate fine-tuning for each task, Omni-DNA leverages auto-regressive transformer-based training and multi-task fine-tuning, enabling a single model to perform a wide range of genomic tasks with state-of-the-art performance. Omni-DNA models range from 20M to 1B parameters and support tasks such as sequence annotation, regulatory element classification, acetylation/methylation prediction, and DNA2Function/DNA2Image mapping. | Size | Training Tokens | Layers | Hidden Size | Attention Heads | Context Length | |-------|----------------|--------|-------------|-----------------|----------------| | Omni-DNA 20M | 300B | 8 | 256 | 8 | 250 | | Omni-DNA 60M | 300B | 8 | 512 | 8 | 250 | | Omni-DNA 116M | 300B | 12 | 768 | 16 | 250 | | Omni-DNA 300M | 300B | 16 | 1024 | 16 | 250 | | Omni-DNA 700M | 300B | 16 | 1536 | 16 | 250 | | Omni-DNA 1B | 300B | 16 | 2048 | 16 | 250 | - Supported by: Microsoft Research Asia - Model type: Auto-regressive transformer-based genomic model - License: mit - Date cutoff: 2024 - Contact: Research inquiries at `[email protected]` - Paper: Omni-DNA: Scaling Auto-Regressive Transformer to Multi-Tasking Genomic Foundation Model - Codebase: https://github.com/Zehui127/Omni-DNA - Dataset: Pretrained on 300B nucleotides from multi-species genome datasets Omni-DNA is trained to perform multiple genomic tasks including: - Regulatory Element Classification: Enhancer/promoter/splice site detection - Histone Modification Prediction: Acetylation and methylation state identification - Genomic Function Annotation: DNA-to-text mapping (DNA2Function) - Cross-modal Learning: DNA-to-image mapping (DNA2Image) - Multi-task Learning: A single model can solve multiple tasks simultaneously Supervised Finetuning (Make Prediction in the Generative Manner)

Omni-DNA is a cross-modal, multi-task genomic foundation model designed to generalize across diverse genomic tasks. Unlike previous Genomic Foundation Models (GFMs), which require separate fine-tuning for each task, Omni-DNA leverages auto-regressive transformer-based training and multi-task fine-tuning, enabling a single model to perform a wide range of genomic tasks with state-of-the-art performance. Omni-DNA models range from 20M to 1B parameters and support tasks such as sequence annotation, regulatory element classification, acetylation/methylation prediction, and DNA2Function/DNA2Image mapping. | Size | Training Tokens | Layers | Hidden Size | Attention Heads | Context Length | |-------|----------------|--------|-------------|-----------------|----------------| | Omni-DNA 20M | 300B | 8 | 256 | 8 | 250 | | Omni-DNA 60M | 300B | 8 | 512 | 8 | 250 | | Omni-DNA 116M | 300B | 12 | 768 | 16 | 250 | | Omni-DNA 300M | 300B | 16 | 1024 | 16 | 250 | | Omni-DNA 700M | 300B | 16 | 1536 | 16 | 250 | | Omni-DNA 1B | 300B | 16 | 2048 | 16 | 250 | - Supported by: Microsoft Research Asia - Model type: Auto-regressive transformer-based genomic model - License: mit - Date cutoff: 2024 - Contact: Research inquiries at `[email protected]` - Paper: Omni-DNA: Scaling Auto-Regressive Transformer to Multi-Tasking Genomic Foundation Model - Codebase: https://github.com/Zehui127/Omni-DNA - Dataset: Pretrained on 300B nucleotides from multi-species genome datasets Omni-DNA is trained to perform multiple genomic tasks including: - Regulatory Element Classification: Enhancer/promoter/splice site detection - Histone Modification Prediction: Acetylation and methylation state identification - Genomic Function Annotation: DNA-to-text mapping (DNA2Function) - Cross-modal Learning: DNA-to-image mapping (DNA2Image) - Multi-task Learning: A single model can solve multiple tasks simultaneously Supervised Finetuning (Make Prediction in the Generative Manner)