turboderp

This is just a quick mix of the 2.25 bpw quant with attention, dense layers and shared experts in 4.0 bpw.

⚠️ Requires ExLlamaV3 v0.0.7 (or v0.0.6 `dev` branch) 2.00 bits per weight 3.00 bits per weight 4.00 bits per weight 5.00 bits per weight 2.08 bits per weight 2.27 bits per weight 2.78 bits per weight 3.14 bits per weight 3.53 bits per weight 4.06 bits per weight 4.51 bits per weight

⚠️ Requires ExLlamaV3 v0.0.7 (or v0.0.6 `dev` branch) 2.00 bits per weight 3.00 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight 2.08 bits per weight 2.27 bits per weight 2.78 bits per weight 3.14 bits per weight 3.53 bits per weight 4.06 bits per weight 4.51 bits per weight

2.00 bits per weight 2.50 bits per weight 3.00 bits per weight 3.50 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight 8.00 bits per weight / H8

2.00 bits per weight 2.50 bits per weight 3.00 bits per weight 3.50 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight 8.00 bits per weight / H8

EXL3 quants of Grok-3-reasoning-gemma3-12B-distilled-HF

⚠️ Requires ExLlamaV3 v0.0.12 (or v0.0.11 `dev` branch) 2.00 bits per weight 3.00 bits per weight 4.00 bits per weight 2.04 bits per weight 2.27 bits per weight 3.04 bits per weight 3.50 bits per weight 4.03 bits per weight . | KL-div | ppl | HumanEval@1 ---------|--------|-------|------------- 2.00 bpw | 0.400 | 10.92 | 80.5% 2.04 bpw | 0.297 | 10.23 | 87.1% 2.27 bpw | 0.252 | 9.78 | 88.4% 3.00 bpw | 0.141 | 8.99 | 87.8% 3.04 bpw | 0.117 | 8.73 | 87.2% 3.50 bpw | 0.094 | 8.78 | 88.4% 4.00 bpw | 0.087 | 8.58 | 89.6% 4.03 bpw | 0.077 | 8.61 | 87.8% original | - | 8.51 | 87.2%¹

2.00 bits per weight 2.50 bits per weight 3.00 bits per weight 3.50 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight . | MMLU | 95% CI ----------|--------------|------------ 2.0 bpw | 58.90% | +/- 1.50% 2.5 bpw | 64.20% | +/- 1.46% 3.0 bpw | 67.00% | +/- 1.43% 3.5 bpw | 67.70% | +/- 1.43% 4.0 bpw | 69.40% | +/- 1.40% 5.0 bpw | 70.30% | +/- 1.39% 6.0 bpw | 69.60% | +/- 1.40%

2.00 bits per weight 2.50 bits per weight 3.00 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight 8.00 bits per weight / H8

2.00 bits per weight 2.50 bits per weight 3.00 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight 8.00 bits per weight / H8

2.00 bits per weight 2.50 bits per weight 3.00 bits per weight 3.50 bits per weight 4.00 bits per weight 5.00 bits per weight 6.00 bits per weight 8.00 bits per weight / H8

2.00 bits per weight 2.25 bits per weight 2.50 bits per weight 3.00 bits per weight

2.07 bits per weight 2.50 bits per weight 3.00 bits per weight 4.00 bits per weight 5.00 bits per weight