Snowflake

--- pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - mteb - arctic - snowflake-arctic-embed - transformers.js license: apache-2.0 language: - af - ar - az - be - bg - bn - ca - ceb - cs - cy - da - de - el - en - es - et - eu - fa - fi - fr - gl - gu - he - hi - hr - ht - hu - hy - id - is - it - ja - jv - ka - kk - km - kn - ko - ky - lo - lt - lv - mk - ml - mn - mr - ms - my - ne - nl - pa - pl - pt - qu - ro - ru - si - sk - sl - so

--- pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - mteb - arctic - snowflake-arctic-embed - transformers.js model-index: - name: snowflake-arctic-embed-m results: - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 76.80597014925374 - type: ap value: 39.3119815

News | Models | Usage | Evaluation | Contact | FAQ License | Acknowledgement 12/04/2024: Release of snowflake-arctic-embed-l-v2.0 and snowflake-arctic-embed-m-v2.0 our newest models with multilingual workloads in mind. These models outperform prior versions of Arctic Embed and we suggest these replace prior versions! 07/26/2024: Release preprint [[2407.18887] Embedding And Clustering Your Data Can Improve Contrastive Pretraining](https://arxiv.org/abs/2407.18887) on arXiv. 07/18/2024: Release of `snowflake-arctic-embed-m-v1.5`, capable of producing highly compressible embedding vectors that preserve quality even when squished as small as 128 bytes per vector. Details about the development of this model are available in the launch post on the Snowflake engineering blog. 05/10/2024: Release the technical report on Arctic Embed 04/16/2024: Release the snowflake-arctic-embed family of text embedding models. The releases are state-of-the-art for Retrieval quality at each of their representative size profiles. [Technical Report]() is coming shortly. For more details, please refer to our Github: Arctic-Text-Embed. snowflake-arctic-embed is a suite of text embedding models that focuses on creating high-quality retrieval models optimized for performance. The `snowflake-arctic-embedding` models achieve state-of-the-art performance on the MTEB/BEIR leaderboard for each of their size variants. Evaluation is performed using these scripts. As shown below, each class of model size achieves SOTA retrieval accuracy compared to other top models. The models are trained by leveraging existing open-source text representation models, such as bert-base-uncased, and are trained in a multi-stage pipeline to optimize their retrieval performance. First, the models are trained with large batches of query-document pairs where negatives are derived in-batch—pretraining leverages about 400m samples of a mix of public datasets and proprietary web search data. Following pretraining models are further optimized with long training on a smaller dataset (about 1m samples) of triplets of query, positive document, and negative document derived from hard harmful mining. Mining of the negatives and data curation is crucial to retrieval accuracy. A detailed technical report can be found here. | Name | MTEB Retrieval Score (NDCG @ 10) | Parameters (Millions) | Embedding Dimension | | ----------------------------------------------------------------------- | -------------------------------- | --------------------- | ------------------- | | snowflake-arctic-embed-xs | 50.15 | 22 | 384 | | snowflake-arctic-embed-s | 51.98 | 33 | 384 | | snowflake-arctic-embed-m | 54.90 | 110 | 768 | | snowflake-arctic-embed-m-long | 54.83 | 137 | 768 | | snowflake-arctic-embed-l | 55.98 | 335 | 1024 | Aside from being great open-source models, the largest model, snowflake-arctic-embed-l, can serve as a natural replacement for closed-source embedding, as shown below. | Model Name | MTEB Retrieval Score (NDCG @ 10) | | ------------------------------------------------------------------ | -------------------------------- | | snowflake-arctic-embed-l | 55.98 | | Google-gecko-text-embedding | 55.7 | | text-embedding-3-large | 55.44 | | Cohere-embed-english-v3.0 | 55.00 | | bge-large-en-v1.5 | 54.29 | This tiny model packs quite the punch. Based on the all-MiniLM-L6-v2 model with only 22m parameters and 384 dimensions, this model should meet even the strictest latency/TCO budgets. Despite its size, its retrieval accuracy is closer to that of models with 100m paramers. | Model Name | MTEB Retrieval Score (NDCG @ 10) | | ------------------------------------------------------------------- | -------------------------------- | | snowflake-arctic-embed-xs | 50.15 | | GIST-all-MiniLM-L6-v2 | 45.12 | | gte-tiny | 44.92 | | all-MiniLM-L6-v2 | 41.95 | | bge-micro-v2 | 42.56 | Based on the intfloat/e5-small-unsupervised model, this small model does not trade off retrieval accuracy for its small size. With only 33m parameters and 384 dimensions, this model should easily allow scaling to large datasets. | Model Name | MTEB Retrieval Score (NDCG @ 10) | | ------------------------------------------------------------------ | -------------------------------- | | snowflake-arctic-embed-s | 51.98 | | bge-small-en-v1.5 | 51.68 | | Cohere-embed-english-light-v3.0 | 51.34 | | text-embedding-3-small | 51.08 | | e5-small-v2 | 49.04 | Based on the intfloat/e5-base-unsupervised model, this medium model is the workhorse that provides the best retrieval performance without slowing down inference. | Model Name | MTEB Retrieval Score (NDCG @ 10) | | ------------------------------------------------------------------ | -------------------------------- | | snowflake-arctic-embed-m | 54.90 | | bge-base-en-v1.5 | 53.25 | | nomic-embed-text-v1.5 | 53.25 | | GIST-Embedding-v0 | 52.31 | | gte-base | 52.31 | Based on the nomic-ai/nomic-embed-text-v1-unsupervised model, this long-context variant of our medium-sized model is perfect for workloads that can be constrained by the regular 512 token context of our other models. Without the use of RPE, this model supports up to 2048 tokens. With RPE, it can scale to 8192! | Model Name | MTEB Retrieval Score (NDCG @ 10) | | ------------------------------------------------------------------ | -------------------------------- | | snowflake-arctic-embed-m-long | 54.83 | | nomic-embed-text-v1.5 | 53.01 | | nomic-embed-text-v1 | 52.81 | Based on the intfloat/e5-large-unsupervised model, this large model is a direct drop-in for closed APIs and delivers the most accurate retrieval experience. | Model Name | MTEB Retrieval Score (NDCG @ 10) | | ------------------------------------------------------------------ | -------------------------------- | | snowflake-arctic-embed-l | 55.98 | | UAE-Large-V1 | 54.66 | | bge-large-en-v1.5 | 54.29 | | mxbai-embed-large-v1 | 54.39 | | e5-Large-v2 | 50.56 | You can use the sentence-transformers package to use an snowflake-arctic-embed model, as shown below. You can use the transformers package to use an snowflake-arctic-embed model, as shown below. For optimal retrieval quality, use the CLS token to embed each text portion and use the query prefix below (just on the query). If you haven't already, you can install the Transformers.js JavaScript library from NPM by running: You can then use the model to compute embeddings as follows: OpenAI compatible API deployment with Infinity and Docker. Feel free to open an issue or pull request if you have any questions or suggestions about this project. You also can email Daniel Campos([email protected]). Arctic is licensed under the Apache-2. The released models can be used for commercial purposes free of charge. We want to thank the open-source community, which has provided the great building blocks upon which we could make our models. We thank our modeling engineers, Danmei Xu, Luke Merrick, Gaurav Nuti, and Daniel Campos, for making these great models possible. We thank our leadership, Himabindu Pucha, Kelvin So, Vivek Raghunathan, and Sridhar Ramaswamy, for supporting this work. We also thank the open-source community for producing the great models we could build on top of and making these releases possible. Finally, we thank the researchers who created BEIR and MTEB benchmarks. It is largely thanks to their tireless work to define what better looks like that we could improve model performance.

Overview Arctic-Text2SQL-R1-7B is a 7-billion-parameter Text-to-SQL model fine-tuned using Group Relative Policy Optimization (GRPO) with a simple execution-based reward signal. It converts natural language questions into executable SQL queries. Read more in our paper: Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL. - Lightweight RL formulation: Uses only execution correctness and syntax validity as rewards. - State-of-the-art performance: Achieves 68.9% execution accuracy on BIRD-dev and 68.5% on BIRD-test, with an average of 57.2% across six benchmarks (BIRD, Spider, Spider2.0, Spider-DK, EHRSQL, ScienceBenchmark) - Efficiency: Outperforms many 70B+ models with only 7B parameters. - Interactive natural language interfaces to relational databases. - Data analytics tools enabling non-technical users to query databases. Not intended for: - Generation of non-SQL text or free-form natural language tasks. - Production systems without validation, especially in safety-critical domains. | Benchmark | Dev/Test Accuracy | | ---------------- | ----------------- | | BIRD-dev | 68.9% | | BIRD-test | 68.5% | | Spider-test | 88.8% | | Spider2.0-DK | 15.6% | | EHRSQL | 36.7% | | ScienceBenchmark | 51.8% | | Average | 57.2% | Ethical Considerations - Avoid using for private or sensitive data without proper oversight. - Validate generated SQL to prevent data leakage or unauthorized access.

Arctic-TILT is a state-of-the-art, sub-billion parameter model for Document Understanding. It is designed for high efficiency and long-context processing, achieving performance on par with models 1000x its size on a single 24GB GPU. GitHub Repository (Code & Examples): Snowflake-Labs/arctic-tilt Paper: Arctic-TILT: Business Document Understanding at Sub-Billion Scale. Model type: Encoder-Decoder Transformer Parameters: 822M Core Task: Document AI (Question Answering, Key Information Extraction) This model is optimized for use with `vLLM` and requires a custom scheduler and preprocessor. The `TiltPreprocessor` is available in the GitHub repository.

The Snowflake AI Research team is releasing a series of SwiftKV optimized Llama-3.x models. SwiftKV is a series of inference optimizations that goes beyond traditional key-value (KV) cache compression. This method reduces computational overhead during prompt processing by combining model rewiring and knowledge-preserving self-distillation, allowing prefill tokens to skip up to half the model's layers. SwiftKV achieves up to 2x improvements in throughput, latency, and cost efficiency with minimal accuracy loss, making LLM deployments more performant and economically viable. For more details about SwiftKV and how to use it: ❄️ SwiftKV: Accelerating Enterprise LLM Workloads with Knowledge Preserving Compute Reduction (blog) 📝 SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (arXiv) 🚀 Getting started guide release-2508 (Aug 2025): Initial model release (context up to 128K) To evaluate SwiftKV’s performance, we focus on the following key metrics (see more details in our blog): Combined throughput: The total number of input and output tokens processed per second. This determines: For batch processing, the time required to complete jobs. For interactive use, the volume of concurrent requests a system can handle. TTFT: The latency between a user request and receiving the first token in the response. TPOT: The latency between subsequent tokens after the first token. Combined input and output throughput for Llama 3.1 70B (left) and Llama 3.1 405B (right) across a range of input lengths (bottom). TTFT (top) and TPOT (bottom) for input lengths 2000 (left), 8000 (middle), and 32000 (right) for Llama 3.1 405B fp8 model. For each experiment, a range of different request arrival rates is simulated. Each request generates 256 output tokens. For a full breakdown on evaluation metrics and performance impact please refer to our blog and arXiv paper) but below we've outlined some relevant evaluation metrics. | Llama-3.1-405B-Instruct-FP8 | Arc Challenge | Winogrande | HellaSwag | TruthfulQA | MMLU | MMLU cot | GSM8K | Avg | |-----------|---------------|------------|-----------|------------|------|----------|-------|-----| | Baseline | 94.7 | 87.0 | 88.3 | 64.7 | 87.5 | 88.1 | 96.1 | 86.6 | | 50% SingleInputKV | 94.0 | 86.3 | 88.1 | 64.2 | 85.7 | 87.5 | 95.2 | 85.9 | | Llama-3.1-8B-Instruct | Arc Challenge | Winogrande | HellaSwag | TruthfulQA | MMLU | MMLU cot | GSM8K | Avg | |-----------|---------------|------------|-----------|------------|------|----------|-------|-----| | Baseline | 82.00 | 77.90 | 80.40 | 54.56 | 67.90 | 70.63 | 82.56 | 73.71 | | 50% SingleInputKV | 80.38 | 78.22 | 79.30 | 54.54 | 67.30 | 69.73 | 79.45 | 72.70 | Instructions on how to use vLLM for both evaluation and performance benchmarks: https://github.com/Snowflake-Labs/vllm/tree/swiftkv/examples/swiftkv