diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2024-10-09 12:54:40 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-10-09 12:54:40 +0300 |
| commit | b30c9e10d8710a49b2d2ab98d086b9f11bfaa228 (patch) | |
| tree | d2d0feb6ca78d3393a88acf81459e2f31d17c93a /ggml/src/ggml-quants.c | |
| parent | c0ddc644bbb53d1fac10cac454756657b5f1ba32 (diff) | |
New SOTA quantization: 4.25 bpw IQ4_KS (#83)
* iq4_k_xxs: basics
* WIP + adding iq3_kl quantization mix
* iq4_xxs: this looks very viable compared to iq4_xs
At the same 4.25 bpw PPL is always better, for some models
significantly better. I'll rename to iq4_ks and keep it.
* iq4_xxs: CUDA dot product
We get TG-128 = 126 t/s for LLaMA-3.1-8B, compared to 123 t/s for q4_0.
* iq4_xxs: scalar CPU dot product
Also fix the breakage I caused with the dedicated work buffer
quantization portion when the multiplication is not done
via iqk_mul_mat.
* iq4_xxs: Zen4
I noticed that iq4_xs is wrong on Zen4 (and possibly AVX2).
Again the same mistake of packing int32_t back to int16_t,
which overflows occasionally (just occasionally, that's why the
result doesn't look completely wrong, so I didn't notice).
* Fix iq4_xs (Zen4)
* iq4_xxs: AVX2
* iq4_xxs: ARM_NEON
* iq4_xxs: Metal
* iq4_xxs: slightly faster TG on Metal
* iq4_xxs: rename to iq4_ks
After all, tt is a smaller variant of iq4_k.
* iq3_kl: use iq4_ks instead of iq4_k/iq4_xs
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/ggml-quants.c')
| -rw-r--r-- | ggml/src/ggml-quants.c | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/ggml/src/ggml-quants.c b/ggml/src/ggml-quants.c index f5fff22e..40978ac0 100644 --- a/ggml/src/ggml-quants.c +++ b/ggml/src/ggml-quants.c @@ -14947,7 +14947,7 @@ bool ggml_validate_row_data(enum ggml_type type, const void * data, size_t nbyte return false; } - if (type != GGML_TYPE_IQ2_TN && type != GGML_TYPE_IQ1_TN && nbytes % ggml_type_size(type) != 0) { + if (type != GGML_TYPE_IQ2_TN && type != GGML_TYPE_IQ1_TN && type != GGML_TYPE_IQ4_KS && nbytes % ggml_type_size(type) != 0) { fprintf(stderr, "%s: invalid size %zu for type %s (type size = %zu)\n", __func__, nbytes, ggml_type_name(type), ggml_type_size(type)); return false; } @@ -15166,6 +15166,7 @@ bool ggml_validate_row_data(enum ggml_type type, const void * data, size_t nbyte case GGML_TYPE_IQ6_K: break; case GGML_TYPE_IQ2_TN: break; case GGML_TYPE_IQ1_TN: break; + case GGML_TYPE_IQ4_KS: break; case GGML_TYPE_Q4_0_4_4: case GGML_TYPE_Q4_0_4_8: { |