From 76b97c80645362ac65a2e33043fd8d46bdaf8c56 Mon Sep 17 00:00:00 2001 From: Kawrakow Date: Wed, 16 Oct 2024 15:18:26 +0300 Subject: Adding IQ4_KSS: 4.0 bpw quants (#89) * iq4_kss: WIP * iq4_kss: CUDA dequantize works So we can run perplexity. Sadly, the result does not look good on the bpw vs quantization error plot. * iq4_kss: slightly better quantization * iq4_kss: another small quantization improvement * iq4_kss: CUDA works TG-128 performance is very decent with 131 t/s for LLaMA-3.1-8B. In comparison, we have 123 t/s for q4_0 and 128 t/s for iq4_ks. I.e., the reduced model size more than offsets the additional bit fiddling required for iq4_kss. * iq4_kss: new bit arrangement - CUDA and Zen4 work Did not lose performance on CUDA. Zen4 is decent, but not great: PP-512(LLaMA-3.1-8B) = 163 t/s. TG-128 is of course better than other 4-bit quants due to smaller model size. We get 14.5 t/s @ 8 threads. * iq4_kss: ARM_NEON. Predictably very slow * iq4_kss: Metal PP is not too bad - just 10% slower than q4_0. But TG is 30% slower, i.e., predictably bad. * iq4_kss: somewhat faster Metal dot product 45.75 t/s -> 48.75 t/s. Still 22% slower than q4_0 * iq4_kss: AVX2 Bad, but better than I expected. PP-512(LLaMA-3.1-8B) = 167 t/s on the Ryzen-5950X. I.e., with 32 AVX2 threads we get the performance of 16 Zen4 threads. * iq4_kss: very slightly faster Metal dot product 48.7 t/s -> 49.3 t/s --------- Co-authored-by: Iwan Kawrakow --- ggml/include/ggml.h | 2 ++ 1 file changed, 2 insertions(+) (limited to 'ggml/include') diff --git a/ggml/include/ggml.h b/ggml/include/ggml.h index fd7c23b9..a467c297 100644 --- a/ggml/include/ggml.h +++ b/ggml/include/ggml.h @@ -405,6 +405,7 @@ extern "C" { GGML_TYPE_IQ1_TN = 143, GGML_TYPE_IQ4_KS = 144, GGML_TYPE_IQ2_KS = 145, + GGML_TYPE_IQ4_KSS = 146, GGML_TYPE_COUNT, }; @@ -462,6 +463,7 @@ extern "C" { GGML_FTYPE_MOSTLY_IQ1_TN = 136, // except 1d tensors GGML_FTYPE_MOSTLY_IQ4_KS = 137, // except 1d tensors GGML_FTYPE_MOSTLY_IQ2_KS = 138, // except 1d tensors + GGML_FTYPE_MOSTLY_IQ4_KSS = 139, // except 1d tensors }; // available tensor operations: -- cgit v1.2.3