chandar-lab

18 models • 1 total models in database

Sort by:

AMPLIFY_120M

license:mit

4,935

AMPLIFY_350M

license:mit

1,938

NeoBERT

NeoBERT is a next-generation encoder model for English text representation, pre-trained from scratch on the RefinedWeb dataset. NeoBERT integrates state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. It is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it is the most efficient model of its kind and achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine-tuning conditions. Ensure you have the following dependencies installed: If you would like to use sequence packing (un-padding), you will need to also install flash-attention: Features | Feature | NeoBERT | |---------------------------|-----------------------------| | `Depth-to-width` | 28 × 768 | | `Parameter count` | 250M | | `Activation` | SwiGLU | | `Positional embeddings` | RoPE | | `Normalization` | Pre-RMSNorm | | `Data Source` | RefinedWeb | | `Data Size` | 2.8 TB | | `Tokenizer` | google/bert | | `Context length` | 4,096 | | `MLM Masking Rate` | 20% | | `Optimizer` | AdamW | | `Scheduler` | CosineDecay | | `Training Tokens` | 2.1 T | | `Efficiency` | FlashAttention | Model weights and code repository are licensed under the permissive MIT license. If you use this model in your research, please cite: For questions, do not hesitate to reach out and open an issue on here or on our GitHub.

license:mit

1,797

184

SaAMPLIFY_350M

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. - Developed by: [More Information Needed] - Funded by [optional]: [More Information Needed] - Shared by [optional]: [More Information Needed] - Model type: [More Information Needed] - Language(s) (NLP): [More Information Needed] - License: [More Information Needed] - Finetuned from model [optional]: [More Information Needed] - Repository: [More Information Needed] - Paper [optional]: [More Information Needed] - Demo [optional]: [More Information Needed] Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). - Hardware Type: [More Information Needed] - Hours used: [More Information Needed] - Cloud Provider: [More Information Needed] - Compute Region: [More Information Needed] - Carbon Emitted: [More Information Needed]

—

735

NovoMolGen_300M_SMILES_BPE

NovoMolGen is a family of molecular foundation models trained on 1.5 billion ZINC-22 molecules with Llama architectures and FlashAttention. It achieves state-of-the-art performance on both unconstrained and goal-directed molecule generation tasks. Transformers-native HF checkpoint (`revision="hf-checkpoint"`) We also publish a Transformers-native checkpoint on the `hf-checkpoint` revision. This version loads directly with `AutoModelForCausalLM` and works out-of-the-box with `.generate(...)`.

llama