RedHatAI

--- tags: - int4 - vllm language: - en - de - fr - it - pt - hi - es - th pipeline_tag: text-generation license: llama3.1 base_model: meta-llama/Meta-Llama-3.1-70B-Instruct ---

--- language: - en - fr - de - es - it - pt - hi - id - tl - vi - ar - bg - zh - da - el - fa - fi - he - ja - ko - ms - nl - no - pl - ro - ru - sr - sv - th - tr - uk - ur - zsm - nld base_model: - mistralai/Mistral-Small-3.1-24B-Instruct-2503 pipeline_tag: image-text-to-text tags: - mistralai - mistral - mistral3 - mistral-small - neuralmagic - redhat - llmcompressor - quantized - W4A16 - INT4 - conversational - compressed-tensors - fast license: apache-2.0 license_name: apache-2.0 name: RedH

--- license: apache-2.0 base_model: mistralai/Mistral-7B-Instruct-v0.3

--- tags: - fp8 - vllm language: - en - de - fr - it - pt - hi - es - th pipeline_tag: text-generation license: llama3.1 base_model: meta-llama/Meta-Llama-3.1-8B-Instruct ---

--- language: - en - de - fr - it - pt - hi - es - th base_model: - meta-llama/Llama-3.3-70B-Instruct pipeline_tag: text-generation tags: - llama - facebook - meta - llama-3 - fp8 - quantized - conversational - text-generation-inference - compressed-tensors license: llama3.3 license_name: llama-3.3 name: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic description: This model was obtained by quantizing activation and weights of Llama-3.3-70B-Instruct to FP8 data type. readme: https://huggingface.co/R

Model Overview - Verifier: openai/gpt-oss-20b - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 11/21/2025 - Version: 2.0 - Model Developers: RedHat This is a speculator model designed for use with openai/gpt-oss-20b, based on the EAGLE-3 speculative decoding algorithm. It was trained using the speculators library on a combination of the Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered dataset and the `trainsft` split of the HuggingFaceH4/ultrachat200k dataset. This model should be used with the openai/gpt-oss-20b chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.63 2.05 2.18 2.31 2.33 2.38 2.35 - temperature: 0.6 - topp: 0.95 - topk: 20 - repetitions: 3 - time per experiment: 10min - hardware: 2xA100 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 Command ```bash GUIDELLMPREFERREDROUTE="chatcompletions" \ guidellm benchmark \ --target "http://localhost:8000/v1" \ --data "RedHatAI/speculatorbenchmarks" \ --data-args '{"datafiles": "HumanEval.jsonl"}' \ --rate-type sweep \ --max-seconds 600 \ --output-path "gpt-oss-20b-HumanEval.json" \ --backend-args '{"extrabody": {"chatcompletions": {"temperature":0.0}}}'

Model Overview - Model Architecture: Qwen2.5-VL-7B-Instruct - Input: Vision-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 2/24/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of Qwen/Qwen2.5-VL-7B-Instruct to FP8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below as part a multimodal announcement blog. The model was evaluated using mistral-evals for vision-related tasks and using lmevaluationharness for select text-based benchmarks. The evaluations were conducted using the following commands: Vision Tasks - vqav2 - docvqa - mathvista - mmmu - chartqa Category Metric Qwen/Qwen2.5-VL-7B-Instruct neuralmagic/Qwen2.5-VL-7B-Instruct-FP8-Dynamic Recovery (%) Vision MMMU (val, CoT) explicitpromptrelaxedcorrectness 52.00 52.55 101.06% ChartQA (test, CoT) anywhereinanswerrelaxedcorrectness 86.44 86.80 100.42% Mathvista (testmini, CoT) explicitpromptrelaxedcorrectness 69.47 71.07 102.31% This model achieves up to 1.3x speedup in single-stream deployment and 1.37x in multi-stream deployment, depending on hardware and use-case scenario. The following performance benchmarks were conducted with vLLM version 0.7.2, and GuideLLM. Single-stream performance (measured with vLLM version 0.7.2) Document Visual Question Answering 1680W x 2240H 64/128 Visual Reasoning 640W x 480H 128/128 Image Captioning 480W x 360H 0/128 Hardware Model Average Cost Reduction Latency (s) Queries Per Dollar Latency (s)th> Queries Per Dollar Latency (s) Queries Per Dollar neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w8a8 1.50 3.6 1248 2.1 2163 2.0 2237 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w4a16 2.05 3.3 1351 1.4 3252 1.4 3321 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w8a8 1.24 2.4 851 1.4 1454 1.3 1512 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w4a16 1.49 2.2 912 1.1 1791 1.0 1950 neuralmagic/Qwen2.5-VL-7B-Instruct-FP8-Dynamic 1.28 1.6 698 0.9 1181 0.9 1219 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w4a16 1.28 1.6 686 0.9 1191 0.9 1228 Use case profiles: Image Size (WxH) / prompt tokens / generation tokens QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025). Multi-stream asynchronous performance (measured with vLLM version 0.7.2) Document Visual Question Answering 1680W x 2240H 64/128 Visual Reasoning 640W x 480H 128/128 Image Captioning 480W x 360H 0/128 Hardware Model Average Cost Reduction Maximum throughput (QPS) Queries Per Dollar Maximum throughput (QPS) Queries Per Dollar Maximum throughput (QPS) Queries Per Dollar neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w8a8 1.41 0.5 2297 2.3 10137 2.5 11472 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w4a16 1.60 0.4 1828 2.7 12254 3.4 15477 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w8a8 1.27 0.8 1639 3.4 6851 3.9 7918 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w4a16 1.21 0.7 1314 3.0 5983 4.6 9206 neuralmagic/Qwen2.5-VL-7B-Instruct-FP8-Dynamic 1.29 1.2 1331 3.8 4109 4.2 4598 neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w4a16 1.28 1.2 1298 3.8 4190 4.2 4573 Use case profiles: Image Size (WxH) / prompt tokens / generation tokens QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).

Model Overview - Model Architecture: Qwen3VLMoeForConditionalGeneration - Input: Text/Image/Video - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 09/28/202510 - Version: 1.0 - Model Developers:: Red Hat Quantized version of Qwen/Qwen3-VL-235B-A22B-Instruct. This model was obtained by quantizing the weights and activations of Qwen/Qwen3-VL-235B-A22B-Instruct to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLMv1 leaderboard task, using lm-evaluation-harness, on reasoning tasks using lighteval and on vision tasks using lmms-eval. vLLM was used for all evaluations. Category Metric Qwen3-VL-235B-A22B-Instruct Qwen3-VL-235B-A22B-Instruct-FP8-dynamic Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 76.54 75.94 99.2

Model Overview - Model Architecture: Qwen3ForCausalLM - Input: Text - Output: Text - Model Optimizations: - Activation quantization: FP8 - Weight quantization: FP8 - Intended Use Cases: - Reasoning. - Function calling. - Subject matter experts via fine-tuning. - Multilingual instruction following. - Translation. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). - Release Date: 05/02/2025 - Version: 1.0 - Model Developers: RedHat (Neural Magic) This model was obtained by quantizing activations and weights of Qwen3-32B to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. Creation details This model was created with llm-compressor by running the code snippet below. The model was evaluated on the OpenLLM leaderboard tasks (versions 1 and 2), using lm-evaluation-harness, and on reasoning tasks using lighteval. vLLM was used for all evaluations.

Model Overview - Model Architecture: Meta-Llama-3 - Input: Text - Output: Text - Model Optimizations: - Activation quantization: INT8 - Weight quantization: INT8 - Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Meta-Llama-3.1-8B-Instruct, this models is intended for assistant-like chat. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). - Release Date: 7/11/2024 - Version: 1.0 - Validated on: RHOAI 2.20, RHAIIS 3.0, RHELAI 1.5 - License(s): Llama3.1 - Model Developers: Neural Magic This model is a quantized version of Meta-Llama-3.1-8B-Instruct. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model, including multiple-choice, math reasoning, and open-ended text generation. Meta-Llama-3.1-8B-Instruct-quantized.w8a8 achieves 105.4% recovery for the Arena-Hard evaluation, 100.3% for OpenLLM v1 (using Meta's prompting when available), 101.5% for OpenLLM v2, 99.7% for HumanEval pass@1, and 98.8% for HumanEval+ pass@1. This model was obtained by quantizing the weights of Meta-Llama-3.1-8B-Instruct to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between INT8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric dynamic per-token scheme, computing a linear scaling factor at runtime for each token between INT8 and floating point representations. The GPTQ algorithm is applied for quantization, as implemented in the llm-compressor library. GPTQ used a 1% damping factor and 256 sequences of 8,192 random tokens. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. ​​See Red Hat AI Inference Server documentation for more details. See Red Hat Enterprise Linux AI documentation for more details. See Red Hat Openshift AI documentation for more details. This model was created by using the llm-compressor library as presented in the code snipet below. This model was evaluated on the well-known Arena-Hard, OpenLLM v1, OpenLLM v2, HumanEval, and HumanEval+ benchmarks. In all cases, model outputs were generated with the vLLM engine. Arena-Hard evaluations were conducted using the Arena-Hard-Auto repository. The model generated a single answer for each prompt form Arena-Hard, and each answer was judged twice by GPT-4. We report below the scores obtained in each judgement and the average. OpenLLM v1 and v2 evaluations were conducted using Neural Magic's fork of lm-evaluation-harness (branch llama3.1instruct). This version of the lm-evaluation-harness includes versions of MMLU, ARC-Challenge and GSM-8K that match the prompting style of Meta-Llama-3.1-Instruct-evals and a few fixes to OpenLLM v2 tasks. HumanEval and HumanEval+ evaluations were conducted using Neural Magic's fork of the EvalPlus repository. Detailed model outputs are available as HuggingFace datasets for Arena-Hard, OpenLLM v2, and HumanEval. Note: Results have been updated after Meta modified the chat template. Meta-Llama-3.1-8B-Instruct-quantized.w8a8 (this model) The results were obtained using the following commands:

Model Overview - Model Architecture: Qwen/Qwen3-VL-235B-A22B-Instruct - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 10/29/2025 - Version: 1.0 - Model Developers: RedHatAI This model is a quantized version of Qwen/Qwen3-VL-235B-A22B-Instruct. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of Qwen/Qwen3-VL-235B-A22B-Instruct to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 25%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2 and HumanEval64 benchmarks using lm-evaluation-harness. The Reasoning evals were done using ligheval. Category Metric Qwen/Qwen3-VL-235B-A22B-Instruct RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4 (this model) Recovery The results were obtained using the following commands:

Model Overview - Model Architecture: Llama - Input: Text - Output: Text - Model Optimizations: - Activation quantization: INT8 - Weight quantization: INT8 - Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Llama-3.3-70B-Instruct, this models is intended for assistant-like chat. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). - Release Date: 01/20/2025 - Version: 1.0 - Validated on: RHOAI 2.20, RHAIIS 3.0, RHELAI 1.5 - Model Developers: Neural Magic Quantized version of Llama-3.3-70B-Instruct. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model, including multiple-choice, math reasoning, and open-ended text generation. Llama-3.3-70B-Instruct-quantized.w8a8 achieves 99.4% recovery for OpenLLM v1 (using Meta's prompting when available) and 100% for both HumanEval and HumanEval+ pass@1. This model was obtained by quantizing the weights and activations of Llama-3.3-70B-Instruct to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between INT8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric dynamic per-token scheme, computing a linear scaling factor at runtime for each token between INT8 and floating point representations. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. ​​See Red Hat AI Inference Server documentation for more details. See Red Hat Enterprise Linux AI documentation for more details. See Red Hat Openshift AI documentation for more details. This model was created by using the llm-compressor library as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2, HumanEval, and HumanEval+ benchmarks. In all cases, model outputs were generated with the vLLM engine. OpenLLM v1 and v2 evaluations were conducted using Neural Magic's fork of lm-evaluation-harness (branch llama3.1instruct). This version of the lm-evaluation-harness includes versions of MMLU, ARC-Challenge and GSM-8K that match the prompting style of Meta-Llama-3.1-Instruct-evals and a few fixes to OpenLLM v2 tasks. HumanEval and HumanEval+ evaluations were conducted using Neural Magic's fork of the EvalPlus repository. The results were obtained using the following commands:

Model Overview - Model Architecture: google/gemma-3-27b-it - Input: Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Activation quantization: FP16 - Release Date: 6/4/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights of google/gemma-3-27b-it to INT4 data type, ready for inference with vLLM >= 0.8.0. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below: The model was evaluated using lmevaluationharness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands: Category Metric google/gemma-3-27b-it RedHatAI/gemma-3-27b-it-quantized.w8a8 Recovery (%)

Model Overview - Model Architecture: Qwen2 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Intended Use Cases: Intended for commercial and research use in English. Similarly to Meta-Llama-3-8B-Instruct, this models is intended for assistant-like chat. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 6/14/2024 - Version: 1.0 - License(s): apache-2.0 - Model Developers: Neural Magic Quantized version of Qwen2-7B-Instruct. It achieves an average score of 69.44 on the OpenLLM benchmark (version 1), whereas the unquantized model achieves 69.55. This model was obtained by quantizing the weights and activations of Qwen2-7B-Instruct to FP8 data type, ready for inference with vLLM >= 0.5.0. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations. AutoFP8 is used for quantization with 512 sequences of UltraChat. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying AutoFP8 with calibration samples from ultrachat, as presented in the code snipet below. Although AutoFP8 was used for this particular model, Neural Magic is transitioning to using llm-compressor which supports several quantization schemes and models not supported by AutoFP8. The model was evaluated on the OpenLLM leaderboard tasks (version 1) with the lm-evaluation-harness (commit 383bbd54bc621086e05aa1b030d8d4d5635b25e6) and the vLLM engine, using the following command:

Model Overview - Verifier: Qwen/Qwen3-8B - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 07/27/2025 - Version: 1.0 - Model Developers: RedHat This is a speculator model designed for use with Qwen/Qwen3-8B, based on the EAGLE-3 speculative decoding algorithm. It was trained using the speculators library on a combination of the Aeala/ShareGPTVicunaunfiltered and the HuggingFaceH4/ultrachat200k datasets. The model was trained with thinking turned disabled. This model should be used with the Qwen/Qwen3-8B chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.62 1.96 2.13 2.24 2.25 2.29 2.30 - temperature: 0.6 - topp: 0.95 - topk: 20 - repetitions: 3 - time per experiment: 10min - hardware: 1xA100 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 Command ```bash GUIDELLMPREFERREDROUTE="chatcompletions" \ guidellm benchmark \ --target "http://localhost:8000/v1" \ --data "RedHatAI/SpeculativeDecoding" \ --rate-type sweep \ --max-seconds 600 \ --output-path "Qwen3-8B-HumanEval.json" \ --backend-args '{"extrabody": {"chatcompletions": {"temperature":0.6, "topp":0.95, "topk":20}}}'

Model Overview - Model Architecture: Llama-3 - Input: Text - Output: Text - Model Optimizations: - Activation quantization: INT8 - Weight quantization: INT8 - Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Llama-3.2-1B-Instruct, this models is intended for assistant-like chat. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). - Release Date: 9/25/2024 - Version: 1.0 - License(s): Llama3.2 - Model Developers: Neural Magic Quantized version of Llama-3.2-1B-Instruct. It achieves scores within 5% of the scores of the unquantized model for MMLU, ARC-Challenge, GSM-8k, Hellaswag, Winogrande and TruthfulQA. This model was obtained by quantizing the weights of Llama-3.2-1B-Instruct to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between INT8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric dynamic per-token scheme, computing a linear scaling factor at runtime for each token between INT8 and floating point representations. The SmoothQuant algorithm is used to alleviate outliers in the activations, whereas rhe GPTQ algorithm is applied for quantization. Both algorithms are implemented in the llm-compressor library. GPTQ used a 1% damping factor and 512 sequences sequences taken from Neural Magic's LLM compression calibration dataset. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by using the llm-compressor library as presented in the code snipet below. The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA. Evaluation was conducted using the Neural Magic fork of lm-evaluation-harness (branch llama3.1instruct) and the vLLM engine. This version of the lm-evaluation-harness includes versions of MMLU, ARC-Challenge and GSM-8K that match the prompting style of Meta-Llama-3.1-Instruct-evals. The results were obtained using the following commands:

Model Overview - Verifier: meta-llama/Llama-3.1-8B-Instruct - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 07/27/2025 - Version: 1.0 - Model Developers: RedHat This is a speculator model designed for use with meta-llama/Llama-3.1-8B-Instruct, based on the EAGLE-3 speculative decoding algorithm. It was trained using the speculators library on a combination of the Aeala/ShareGPTVicunaunfiltered and the HuggingFaceH4/ultrachat200k datasets. This model should be used with the meta-llama/Llama-3.1-8B-Instruct chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.70 2.19 2.50 2.78 2.77 2.98 2.99 - temperature: 0.6 - topp: 0.9 - repetitions: 5 - time per experiment: 3min - hardware: 1xA100 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 Command ```bash GUIDELLMPREFERREDROUTE="chatcompletions" \ guidellm benchmark \ --target "http://localhost:8000/v1" \ --data "RedHatAI/SpeculativeDecoding" \ --rate-type sweep \ --max-seconds 180 \ --output-path "Llama-3.1-8B-Instruct-HumanEval.json" \ --backend-args '{"extrabody": {"chatcompletions": {"temperature":0.0}}}'

Model Overview - Model Architecture: VoxtralForConditionalGeneration - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Intended Use Cases: Voxtral builds upon Ministral-3B with powerful audio understanding capabilities. - Dedicated transcription mode: Voxtral can operate in a pure speech transcription mode to maximize performance. By default, Voxtral automatically predicts the source audio language and transcribes the text accordingly - Long-form context: With a 32k token context length, Voxtral handles audios up to 30 minutes for transcription, or 40 minutes for understanding - Built-in Q&A and summarization: Supports asking questions directly through audio. Analyze audio and generate structured summaries without the need for separate ASR and language models - Natively multilingual: Automatic language detection and state-of-the-art performance in the world’s most widely used languages (English, Spanish, French, Portuguese, Hindi, German, Dutch, Italian) - Function-calling straight from voice: Enables direct triggering of backend functions, workflows, or API calls based on spoken user intents - Highly capable at text: Retains the text understanding capabilities of its language model backbone, Ministral-3B - Release Date: 08/21/2025 - Version: 1.0 - Model Developers: Red Hat This model was obtained by quantizing activation and weights of Voxtral-Small-24B-2507 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the MLP operators within transformers blocks of the language model are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization. 2. Send requests to the server, according to the use case. See the following examples. This model was quantized using the llm-compressor library as shown below. After quantization, the model can be converted back into the mistralai format using the `convertvoxtralhftomistral.py` script included with the model. The model was evaluated on the Fleurs transcription task. Recovery is computed with respect to the complement of the word error rate (WER). Benchmark Language Voxtral-Small-24B-2507 Voxtral-Small-24B-2507-FP8-dynamic (this model) Recovery

Model Overview - Model Architecture: unsloth/Mistral-Small-3.2-24B-Instruct-2506 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 10/29/2025 - Version: 1.0 - Model Developers: RedHatAI This model is a quantized version of unsloth/Mistral-Small-3.2-24B-Instruct-2506. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of unsloth/Mistral-Small-3.2-24B-Instruct-2506 to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 25%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2 and HumanEval64 benchmarks using lm-evaluation-harness. Category Metric unsloth/Mistral-Small-3.2-24B-Instruct-2506 RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4 Recovery The results were obtained using the following commands:

Model Overview - Model Architecture: NemotronHForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 9/30/2025 - V...

Model Overview - Model Architecture: Qwen/Qwen3-30B-A3B - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 10/29/2025 - Version: 1.0 - Model Developers: RedHatAI This model is a quantized version of Qwen/Qwen3-30B-A3B. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of Qwen/Qwen3-30B-A3B to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2 and HumanEval64 benchmarks using lm-evaluation-harness. The Reasoning evals were done using ligheval. Category Metric Qwen3-30B-A3B Qwen3-30B-A3B-NVFP4 (this model) Recovery The results were obtained using the following commands:

Model Overview - Model Architecture: gpt-oss-120b-BF16 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 08/13/2025 - Version: 1.0 - Model Developers: RedHatAI

Model Overview - Model Architecture: whisper-large-v3 - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of openai/whisper-large-v3 to INT4 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands:

Model Overview - Model Architecture: google/gemma-3-1b-it - Input: Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT8 - Activation quantization: INT8 - Release Date: 6/4/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights of google/gemma-3-1b-it to INT8 data type, ready for inference with vLLM >= 0.8.0. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below: The model was evaluated using lmevaluationharness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands: Category Metric google/gemma-3-1b-it RedHatAI/gemma-3-1b-it-quantized.w8a8 Recovery (%)

Model Overview - Model Architecture: Qwen/Qwen3-235B-A22B-Instruct-2507 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 10/29/2025 - Version: 1.0 - Model Developers: RedHatAI This model is a quantized version of Qwen/Qwen3-235B-A22B-Instruct-2507. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of Qwen/Qwen3-235B-A22B-Instruct-2507 to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2 and HumanEval64 benchmarks using lm-evaluation-harness. The Reasoning evals were done using ligheval. Category Metric Qwen/Qwen3-235B-A22B-Instruct-2507 RedHatAI/Qwen3-235B-A22B-Instruct-2507-NVFP4 (this model) Recovery The results were obtained using the following commands:

Model Overview - Verifier: Qwen/Qwen3-32B - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 09/17/2025 - Version: 1.0 - Model Developers: RedHat This is a speculator model designed for use with Qwen/Qwen3-32B, based on the EAGLE-3 speculative decoding algorithm. It was trained using the speculators library on a combination of the Aeala/ShareGPTVicunaunfiltered and the `trainsft` split of HuggingFaceH4/ultrachat200k datasets. This model should be used with the Qwen/Qwen3-32B chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.62 1.95 2.15 2.23 2.27 2.32 2.33 - temperature: 0.6 - topp: 0.95 - topk: 20 - repetitions: 3 - time per experiment: 10min - hardware: 2xA100 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 Command ```bash GUIDELLMPREFERREDROUTE="chatcompletions" \ guidellm benchmark \ --target "http://localhost:8000/v1" \ --data "RedHatAI/speculatorbenchmarks" \ --data-args '{"datafiles": "HumanEval.jsonl"}' \ --rate-type sweep \ --max-seconds 600 \ --output-path "Qwen3-32B-HumanEval.json" \ --backend-args '{"extrabody": {"chatcompletions": {"temperature":0.6, "topp":0.95, "topk":20}}}'

Model Overview - Model Architecture: MistralForCausalLM - Input: Text - Output: Text - Model Optimizations: - Activation quantization: INT8 - Weight quantization: INT8 - Release Date: 08/29/2025 - Version: 1.0 - Model Developers: Red Hat (Neural Magic) This model was obtained by quantizing weights and activations of Devstral-Small-2507 to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%). Weight quantization also reduces disk size requirements by approximately 50%. This model was created with llm-compressor by running the code snippet below. This model can be deployed efficiently using the vLLM backend, as shown in the example below. The model was evaluated on popular coding tasks (HumanEval, HumanEval+, MBPP, MBPP+) via EvalPlus and vllm backend (v0.10.1.1). For evaluations, we run greedy sampling and report pass@1. The command to reproduce evals: | | Recovery (%) | mistralai/Devstral-Small-2507 | RedHatAI/Devstral-Small-2507-quantized.w8a8 (this model) | | --------------------------- | :----------: | :------------------: | :--------------------------------------------------: | | HumanEval | 100.67 | 89.0 | 89.6 | | HumanEval+ | 101.48 | 81.1 | 82.3 | | MBPP | 98.71 | 77.5 | 76.5 | | MBPP+ | 102.42 | 66.1 | 67.7 | | Average Score | 100.77 | 78.43 | 79.03 |

Model Overview - Model Architecture: Qwen3ForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Intended Use Cases: - Reasoning. - Function calling. - Subject matter experts via fine-tuning. - Multilingual instruction following. - Translation. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). - Release Date: 05/05/2025 - Version: 1.0 - Model Developers: RedHat (Neural Magic) This model was obtained by quantizing the weights of Qwen3-14B to INT4 data type. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights of the linear operators within transformers blocks are quantized. Weights are quantized using a asymmetric per-group scheme, with group size 64. The GPTQ algorithm is applied for quantization, as implemented in the llm-compressor library. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. Creation details This model was created with llm-compressor by running the code snippet below. The model was evaluated on the OpenLLM leaderboard tasks (versions 1 and 2), using lm-evaluation-harness, and on reasoning tasks using lighteval. vLLM was used for all evaluations.

Model Overview - Model Architecture: whisper-large-v3 - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of openai/whisper-large-v3 to FP8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands:

Model Overview - Verifier: meta-llama/Llama-3.3-70B-Instruct - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 09/15/2025 - Version: 1.0 - Model Developers: RedHat This is a speculator model designed for use with meta-llama/Llama-3.3-70B-Instruct, based on the EAGLE-3 speculative decoding algorithm. It was trained using the speculators library on a combination of the Aeala/ShareGPTVicunaunfiltered and the `trainsft` split of HuggingFaceH4/ultrachat200k datasets. This model should be used with the meta-llama/Llama-3.3-70B-Instruct chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.71 2.21 2.52 2.74 2.83 2.87 2.89 - temperature: 0 - repetitions: 5 - time per experiment: 4min - hardware: 4xA100 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 Command ```bash GUIDELLMPREFERREDROUTE="chatcompletions" \ guidellm benchmark \ --target "http://localhost:8000/v1" \ --data "RedHatAI/speculatorbenchmarks" \ --data-args '{"datafiles": "HumanEval.jsonl"}' \ --rate-type sweep \ --max-seconds 240 \ --output-path "Llama-3.3-70B-Instruct-HumanEval.json" \ --backend-args '{"extrabody": {"chatcompletions": {"temperature":0.0}}}'

Model Overview - Model Architecture: whisper-large-v3 - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: INT8 - Activation quantization: INT8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of openai/whisper-large-v3 to INT8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands:

Model Overview - Model Architecture: whisper-large-v3-turbo - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic Quantized version of openai/whisper-large-v3-turbo. This model was obtained by quantizing the weights of openai/whisper-large-v3-turbo to FP8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands:

Model Overview - Model Architecture: Qwen/Qwen3-235B-A22B - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 10/29/2025 - Version: 1.0 - Model Developers: RedHatAI This model is a quantized version of Qwen/Qwen3-235B-A22B. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of Qwen/Qwen3-235B-A22B to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2 and HumanEval64 benchmarks using lm-evaluation-harness. The Reasoning evals were done using ligheval. Category Metric Qwen/Qwen3-235B-A22B RedHatAI/Qwen3-235B-A22B-NVFP4 (this model) Recovery The results were obtained using the following commands:

Model Overview - Model Architecture: Meta-Llama-3.1 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Intended Use Cases: Intended for commercial and research use in multiple languages. Similarly to Meta-Llama-3.1-8B-Instruct, this models is intended for assistant-like chat. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 10/23/2025 - Version: 1.0 - License(s): llama3.1 - Model Developers: RedHatAI This model is a quantized version of Meta-Llama-3.1-8B-Instruct. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of Meta-Llama-3.1-8B-Instruct to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2 and HumanEval64 benchmarks. All evaluations were conducted using lm-evaluation-harness. Category Metric Meta-Llama-3.1-8B-Instruct Llama-3.1-8B-Instruct-NVFP4 (this model) Recovery gsm8kllama 78.17 79.30 101.45 hellaswag 78.43 78.01 99.46 mmlullama 69.37 65.95 95.07 mmlucotllama 72.86 68.60 94.15 truthfulqamc2 55.09 52.95 96.12 winogrande 75.77 74.03 97.70 Average 73.29 71.59 97.68 Category Metric Meta-Llama-3.1-8B-Instruct RedHatAI/Llama-3.1-8B-Instruct-NVFP4 (this model) Recovery (%) The results were obtained using the following commands:

Model Overview - Model Architecture: gemma-3n-E2B-it - Input: Audio-Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT8 - Activation quantization: INT8 - Release Date: 08/01/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights and activations of google/gemma-3n-E2B-it to INT8 data type, ready for inference with vLLM >= 0.10.0 This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated using lmevaluationharness for OpenLLM V1 and V2 text-based benchmarks. The evaluations were conducted using the following commands: Category Metric google/gemma-3n-E2B-it RedHatAI/gemma-3n-E2B-it-quantized.w8a8 Recovery (%)

Model Overview - Model Architecture: Llama4ForConditionalGeneration - Input: Text, Image - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat Quantized version of meta-llama/Llama-4-Scout-17B-16E-Instruct. This model was obtained by quantizing the weights and activations of meta-llama/Llama-4-Scout-17B-16E-Instruct to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLMv1 leaderboard task, using lm-evaluation-harness, on reasoning tasks using lighteval. vLLM was used for all evaluations. Category Metric meta-llama/Llama-4-Scout-17B-16E-Instruct RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-block Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 69.62 68.60 98.53 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 89.09 89.93 100.94

Model Overview - Model Architecture: gemma-3n-E2B-it - Input: Audio-Vision-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 08/01/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights of google/gemma-3n-E2B-it to FP8 data type, ready for inference with vLLM >= 0.10.0 This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated using lmevaluationharness for OpenLLM V1 and V2 text-based benchmarks. The evaluations were conducted using the following commands: Category Metric google/gemma-3n-E2B-it FP8 Dynamic Recovery (%)

Model Overview - Model Architecture: whisper-large-v2 - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of openai/whisper-large-v2 to FP8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands: Fleurs (X→en, WER) cmnhanscn 15.2148 14.7614 103.07%

Model Overview - Model Architecture: Meta-Llama-3.3 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP4 - Activation quantization: FP4 - Intended Use Cases: Intended for commercial and research use in multiple languages. Similarly to Meta-Llama-3.3-8B-Instruct, this models is intended for assistant-like chat. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. - Release Date: 6/25/2025 - Version: 1.0 - License(s): llama3.3 - Model Developers: RedHatAI This model is a quantized version of Meta-Llama-3.3-70B-Instruct. It was evaluated on a several tasks to assess the its quality in comparison to the unquatized model. This model was obtained by quantizing the weights and activations of Meta-Llama-3.3-70B-Instruct to FP4 data type, ready for inference with vLLM>=0.9.1 This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code snipet below. This model was evaluated on the well-known OpenLLM v1, OpenLLM v2, HumanEval, and HumanEval64 benchmarks. All evaluations were conducted using lm-evaluation-harness. Category Metric Meta-Llama-3.3-70B-Instruct RedHatAI/Llama-3.3-70B-Instruct-NVFP4 (this model) Recovery gsm8kllama (8-shot, strict-match) 85.22 77.10 90.47 The results were obtained using the following commands:

Model Overview - Model Architecture: LlamaForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat Quantized version of meta-llama/Llama-3.1-8B-Instruct. This model was obtained by quantizing the weights and activations of meta-llama/Llama-3.1-8B-Instruct to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLMv1 leaderboard task, using lm-evaluation-harness, on reasoning tasks using lighteval. vLLM was used for all evaluations. Category Metric meta-llama/Llama-3.1-8B-Instruct RedHatAI/Llama-3.1-8B-Instruct-FP8-block Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 60.92 60.92 100.00 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 81.89 81.41 99.41

Model Overview - Model Architecture: gemma-3n-E2B-it - Input: Audio-Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Activation quantization: INT16 - Release Date: 08/01/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights of google/gemma-3n-E2B-it to INT4 data type, ready for inference with vLLM >= 0.10.0 This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated using lmevaluationharness for OpenLLM V1 and V2 text-based benchmarks. The evaluations were conducted using the following commands: Category Metric google/gemma-3n-E2B-it RedHatAI/gemma-3n-E2B-it-quantized.w4a16 Recovery (%)

Model Overview - Model Architecture: MistralForCausalLM - Input: Text - Output: Text - Model Optimizations: - Activation quantization: INT4 - Weight quantization: None - Release Date: 08/29/2025 - Version: 1.0 - Model Developers: Red Hat (Neural Magic) This model was obtained by quantizing weights of Devstral-Small-2507 to INT4 data type. This optimization reduces the number of bits used to represent weights from 16 to 4, reducing GPU memory requirements (by approximately 75%). Weight quantization also reduces disk size requirements by approximately 75%. This model can be deployed efficiently using the vLLM backend, as shown in the example below. This model was created with llm-compressor by running the code snippet below. The model was evaluated on popular coding tasks (HumanEval, HumanEval+, MBPP, MBPP+) via EvalPlus and vllm backend (v0.10.1.1). For evaluations, we run greedy sampling and report pass@1. The command to reproduce evals: | | Recovery (%) | mistralai/Devstral-Small-2507 | RedHatAI/Devstral-Small-2507-quantized.w4a16 (this model) | | --------------------------- | :----------: | :------------------: | :--------------------------------------------------: | | HumanEval | 98.65 | 89.0 | 87.8 | | HumanEval+ | 100.0 | 81.1 | 81.1 | | MBPP | 98.97 | 77.5 | 76.7 | | MBPP+ | 102.12 | 66.1 | 67.5 | | Average Score | 99.81 | 78.43 | 78.28 |

Model Overview - Model Architecture: gemma-3n-E4B-it - Input: Audio-Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT8 - Activation quantization: INT8 - Release Date: 08/01/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights and activations of google/gemma-3n-E4B-it to INT8 data type, ready for inference with vLLM >= 0.10.0 This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated using lmevaluationharness for OpenLLM V1 and V2 text-based benchmarks. The evaluations were conducted using the following commands: Category Metric google/gemma-3n-E4B-it RedHatAI/gemma-3n-E4B-it-quantized.w8a8 Recovery (%)

Model Overview - Model Architecture: Qwen3ForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat This model was obtained by quantizing the weights and activations of Qwen/Qwen3-8B to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLM leaderboard task, using lm-evaluation-harness. vLLM was used for all evaluations. Category Metric Qwen/Qwen3-8B nm-testing/Qwen3-8B-FP8-block Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 67.66 67.92 100.38 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 48.56 48.80 100.49

Model Overview - Model Architecture: NemotronHForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Release Date: 10/22/2025 - Version: 1.0 - Model Developers: RedHat (Neural Magic) This model was obtained by quantizing the weights of NVIDIA-Nemotron-Nano-9B-v2 to INT4 data type. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights of the linear operators within transformers blocks are quantized. Weights are quantized using a symmetric per-group scheme, with group size 64. The GPTQ algorithm is applied for quantization, as implemented in the llm-compressor library. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. Creation details This model was created with llm-compressor by running the code snippet below. The model was evaluated on the set of popular reasoning tasks AIME25, Math-500, and GPQA-Diamond, using lighteval `v0.11.1.dev0`. vLLM `v0.11.1rc2.dev191+g80e945298.precompiled` was used as the inference engine for all evaluations. NVIDIA-Nemotron-Nano-9B-v2-quantized.w4a16 (this model)

Model Overview - Verifier: Qwen/Qwen3-14B - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 09/18/2025 - Version: 1.0 - Model Developers: RedHat This is a speculator model designed for use with Qwen/Qwen3-14B, based on the EAGLE-3 speculative decoding algorithm. It was trained using the speculators library on a combination of the Aeala/ShareGPTVicunaunfiltered and the `trainsft` split of HuggingFaceH4/ultrachat200k datasets. This model should be used with the Qwen/Qwen3-14B chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.60 1.90 2.06 2.14 2.17 2.19 2.21 - temperature: 0.6 - topp: 0.95 - topk: 20 - repetitions: 3 - time per experiment: 10min - hardware: 1xA100 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 Command ```bash GUIDELLMPREFERREDROUTE="chatcompletions" \ guidellm benchmark \ --target "http://localhost:8000/v1" \ --data "RedHatAI/speculatorbenchmarks" \ --data-args '{"datafiles": "HumanEval.jsonl"}' \ --rate-type sweep \ --max-seconds 600 \ --output-path "Qwen3-14B-HumanEval.json" \ --backend-args '{"extrabody": {"chatcompletions": {"temperature":0.6, "topp":0.95, "topk":20}}}'

Model Overview - Model Architecture: SmolLM3-3B - Input: Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Activation quantization: None - Release Date: 07/31/2025 - Version: 1.0 - License(s): Apache-2.0 - Model Developers: RedHat (Neural Magic) This model was obtained by quantizing weights of SmolLM3-3B to INT4 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 4, reducing GPU memory requirements (by approximately 75%). Weight quantization also reduces disk size requirements by approximately 75%. Only weights of the linear operators within transformers blocks are quantized. The llm-compressor library is used for quantization. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM aslo supports OpenAI-compatible serving. See the documentation for more details. Creation details This model was created with llm-compressor by running the code snippet below with: This model was evaluated on the well-known reasoning tasks: AIME24, Math-500, and GPQA-Diamond. In all cases, model outputs were generated with the vLLM engine, and evals are collected through LightEval library.

Model Overview - Model Architecture: LlamaForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 06/06/2025 - Version: 1.0 - Intended Use Cases: This model is finetuned to summarize text in the style of Reddit posts. - Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3.1 Community License. - Model Developers: Red Hat (Neural Magic) This model is a quantized version of RedHatAI/Llama-3.1-8B-tldr, which is fine-tuned on the trl-lib/tldr dataset. This model recovers 100% of the BERTScore (0.366) obtained by RedHatAI/Llama-3.1-8B-tldr while providing up to 1.3x speedup. This model can be deployed efficiently using vLLM, as shown in the example below. Run the following command to start the vLLM server: Once your server is started, you can query the model using the OpenAI API: This model was created by applying llm-compressor, as presented in the code snipet below. The model was evaluated on the test split of trl-lib/tldr using the Neural Magic fork of lm-evaluation-harness (tldr branch). One can reproduce these results by using the following command: We evaluated the inference performance of this model using the first 1,000 samples from the training set of the trl-lib/tldr dataset. Benchmarking was conducted with vLLM version `0.9.0.1` and GuideLLM version `0.2.1`. The figure below presents the mean end-to-end latency per request across varying request rates. Results are shown for this model, as well as two variants: - Dense: Llama-3.1-8B-tldr - Sparse-quantized: Sparse-Llama-3.1-8B-tldr-2of4-FP8-dynamic 1. Generate a JSON file containing the first 1,000 training samples: > The average output length is approximately 30 tokens per sample. We capped the generation at 128 tokens to reduce performance skew from rare, unusually verbose completions.

Model Overview - Model Architecture: Qwen3ForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat This model was obtained by quantizing the weights and activations of Qwen/Qwen3-14B to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLM leaderboard task, using lm-evaluation-harness. vLLM was used for all evaluations. Category Metric Qwen/Qwen3-14B nm-testing/Qwen3-14B-FP8-block Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 69.71 69.80 100.12 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 48.56 48.80 100.49

Model Summary: Granite-3.3-8B-Instruct is a 8-billion parameter 128K context length language model fine-tuned for improved reasoning and instruction-following capabilities. Built on top of Granite-3.3-8B-Base, the model delivers significant gains on benchmarks for measuring generic performance including AlpacaEval-2.0 and Arena-Hard, and improvements in mathematics, coding, and instruction following. It supports structured reasoning through \ \ and \ \ tags, providing clear separation between internal thoughts and final outputs. The model has been trained on a carefully balanced combination of permissively licensed data and curated synthetic tasks. - Developers: Granite Team, IBM - Website: Granite Docs - Release Date: April 16th, 2025 - License: Apache 2.0 Supported Languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. However, users may finetune this Granite model for languages beyond these 12 languages. Intended Use: This model is designed to handle general instruction-following tasks and can be integrated into AI assistants across various domains, including business applications. Capabilities Thinking Summarization Text classification Text extraction Question-answering Retrieval Augmented Generation (RAG) Code related tasks Function-calling tasks Multilingual dialog use cases Long-context tasks including long document/meeting summarization, long document QA, etc. Generation: This is a simple example of how to use Granite-3.3-8B-Instruct model. Then, copy the snippet from the section that is relevant for your use case. Comparison with different models over various benchmarks 1 . Scores of AlpacaEval-2.0 and Arena-Hard are calculated with thinking=True Models Arena-Hard AlpacaEval-2.0 MMLU PopQA TruthfulQA BigBenchHard 2 DROP 3 GSM8K HumanEval HumanEval+ IFEval AttaQ Granite-3.1-2B-Instruct 23.3 27.17 57.11 20.55 59.79 61.82 20.99 67.55 79.45 75.26 63.59 84.7 Granite-3.2-2B-Instruct 24.86 34.51 57.18 20.56 59.8 61.39 23.84 67.02 80.13 73.39 61.55 83.23 Granite-3.3-2B-Instruct 28.86 43.45 55.88 18.4 58.97 63.91 44.33 72.48 80.51 75.68 65.8 87.47 Llama-3.1-8B-Instruct 36.43 27.22 69.15 28.79 52.79 73.43 71.23 83.24 85.32 80.15 79.10 83.43 DeepSeek-R1-Distill-Llama-8B 17.17 21.85 45.80 13.25 47.43 67.39 49.73 72.18 67.54 62.91 66.50 42.87 Qwen-2.5-7B-Instruct 25.44 30.34 74.30 18.12 63.06 69.19 64.06 84.46 93.35 89.91 74.90 81.90 DeepSeek-R1-Distill-Qwen-7B 10.36 15.35 50.72 9.94 47.14 67.38 51.78 78.47 79.89 78.43 59.10 42.45 Granite-3.1-8B-Instruct 37.58 30.34 66.77 28.7 65.84 69.87 58.57 79.15 89.63 85.79 73.20 85.73 Granite-3.2-8B-Instruct 55.25 61.19 66.79 28.04 66.92 71.86 58.29 81.65 89.35 85.72 74.31 84.7 Granite-3.3-8B-Instruct 57.56 62.68 65.54 26.17 66.86 69.13 59.36 80.89 89.73 86.09 74.82 88.5 Training Data: Overall, our training data is largely comprised of two key sources: (1) publicly available datasets with permissive license, (2) internal synthetically generated data targeted to enhance reasoning capabilites. Infrastructure: We train Granite-3.3-8B-Instruct using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable and efficient infrastructure for training our models over thousands of GPUs. Ethical Considerations and Limitations: Granite-3.3-8B-Instruct builds upon Granite-3.3-8B-Base, leveraging both permissively licensed open-source and select proprietary data for enhanced performance. Since it inherits its foundation from the previous model, all ethical considerations and limitations applicable to Granite-3.3-8B-Base remain relevant. Resources - ⭐️ Learn about the latest updates with Granite: https://www.ibm.com/granite - 📄 Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/ - 💡 Learn about the latest Granite learning resources: https://github.com/ibm-granite-community/ [1] Evaluated using OLMES (except AttaQ and Arena-Hard scores) [2] Added regex for more efficient asnwer extraction. [3] Modified the implementation to handle some of the issues mentioned here

Model Overview - Model Architecture: gemma-3n-E4B-it - Input: Audio-Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Activation quantization: INT16 - Release Date: 08/01/2025 - Version: 1.0 - Model Developers: RedHatAI This model was obtained by quantizing the weights of google/gemma-3n-E4B-it to INT4 data type, ready for inference with vLLM >= 0.10.0 This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated using lmevaluationharness for OpenLLM V1 and V2 text-based benchmarks. The evaluations were conducted using the following commands: Category Metric google/gemma-3n-E4B-it RedHatAI/gemma-3n-E4B-it-quantized.w4a16 Recovery (%)

Model Overview - Model Architecture: Phi-3-vision-128k-instruct - Input: Vision-Text - Output: Text - Model Optimizations: - Weight quantization: INT4 - Activation quantization: FP16 - Release Date: 1/31/2025 - Version: 1.0 - Model Developers: Neural Magic Quantized version of microsoft/Phi-3-vision-128k-instruct. This model was obtained by quantizing the weights of microsoft/Phi-3-vision-128k-instruct to INT4 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below as part a multimodal announcement blog. This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.

Model Overview - Model Architecture: whisper-small - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of openai/whisper-small to FP8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands: Fleurs (X→en, WER) cmnhanscn 23.0642 24.6761 93.50%

Model Overview - Model Architecture: whisper-large-v3-turbo - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: INT8 - Activation quantization: INT8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic Quantized version of openai/whisper-large-v3-turbo. This model was obtained by quantizing the weights of openai/whisper-large-v3-turbo to INT8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands:

Model Overview - Model Architecture: Llama4ForConditionalGeneration - Input: Text, Image - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat Quantized version of meta-llama/Llama-4-Maverick-17B-128E-Instruct. This model was obtained by quantizing the weights and activations of meta-llama/Llama-4-Maverick-17B-128E-Instruct to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLM leaderboard task, using lm-evaluation-harness. vLLM was used for all evaluations. Category Metric meta-llama/Llama-4-Maverick-17B-128E-Instruct RedHatAI/Llama-4-Maverick-17B-128E-Instruct-block-FP8 Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 73.38 73.38 100.00 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 89.93 90.89 101.07

Model Overview - Model Architecture: LlamaForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat Quantized version of meta-llama/Llama-3.3-70B-Instruct. This model was obtained by quantizing the weights and activations of meta-llama/Llama-3.3-70B-Instruct to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLMv1 leaderboard task, using lm-evaluation-harness, on reasoning tasks using lighteval. vLLM was used for all evaluations. Category Metric meta-llama/Llama-3.3-70B-Instruct nm-testing/Llama-3.3-70B-Instruct-FP8-block Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 72.53 72.61 100.12 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 92.57 92.57 100.00

Model Overview - Model Architecture: Qwen3ForCausalLM - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - Release Date: - Version: 1.0 - Model Developers:: Red Hat This model was obtained by quantizing the weights and activations of Qwen/Qwen3-32B to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. This model was quantized using the llm-compressor library as shown below. The model was evaluated on the OpenLLM leaderboard task, using lm-evaluation-harness. vLLM was used for all evaluations. Category Metric Qwen/Qwen3-32B nm-testing/Qwen3-32B-FP8-block Recovery (%) OpenLLM V1 ARC-Challenge (Acc-Norm, 25-shot) 72.95 72.78 99.77 OpenLLM V2 IFEval (Inst Level Strict Acc, 0-shot) 49.04 49.28 100.49

Llama-4-Maverick-17B-128E-Instruct-speculators.eagle3 Model Overview - Verifier: meta-llama/Llama-4-Maverick-17B-128E-Instruct - Speculative Decoding Algorithm: EAGLE-3 - Model Architecture: Eagle3Speculator - Release Date: 09/17/2025 - Version: 1.0 - Model Developers: RedHat This is a speculator model designed for use with meta-llama/Llama-4-Maverick-17B-128E-Instruct, based on the EAGLE-3 speculative decoding algorithm. It was converted into the speculators format from the model nvidia/Llama-4-Maverick-17B-128E-Eagle3. This model should be used with the meta-llama/Llama-4-Maverick-17B-128E-Instruct chat template, specifically through the `/chat/completions` endpoint. Text Summarization 1.69 2.12 2.37 2.52 2.60 2.63 2.63 - temperature: 0.6 - topp: 0.9 - repetitions: 3 - time per experiment: 3min - hardware: 8xB200 - vLLM version: 0.11.0 - GuideLLM version: 0.3.0 If you use this model, please cite both the original NVIDIA model and the Speculators library: - Original model by NVIDIA Corporation - Conversion and formatting for Speculators/vLLM compatibility - Based on Eagle3 architecture with Llama3 draft head targeting Llama4 verifier

Model Overview - Model Architecture: whisper-small - Input: Audio-Text - Output: Text - Model Optimizations: - Weight quantization: INT8 - Activation quantization: INT8 - Release Date: 04/16/2025 - Version: 1.0 - Model Developers: Neural Magic This model was obtained by quantizing the weights of openai/whisper-small to INT8 data type, ready for inference with vLLM >= 0.5.2. This model can be deployed efficiently using the vLLM backend, as shown in the example below. vLLM also supports OpenAI-compatible serving. See the documentation for more details. This model was created with llm-compressor by running the code snippet below. The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands: Fleurs (X→en, WER) cmnhanscn 23.0642 24.1052 95.68%