diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2025-06-11 11:12:30 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-06-11 11:12:30 +0300 |
| commit | e56061fa12e5fcb7a9a8fa5fca9e74c20d166bcc (patch) | |
| tree | f0627d22fac777db7e397119b2f38421a2b289b8 /ggml/src/ggml.c | |
| parent | 3c1f2c68fdbb9e5be26aad85902c3d86057d69b5 (diff) | |
IQ2_XXS: much faster CPU prompt processing (#515)
* Much faster iq2_xxs GEMM
PP-512 = 290 t/s vs ~110 t/s (iq2_xxs) or 148 t/s (iq2_xxs_r4) on main.
* iq2_xxs: q8_2_x4 GEMM
* iq2_xxs: use template for q8_2_x4 GEMM
* Fix AVX2
* Cleanup
* NEON is not working yet, so still use Q8_K GEMM
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/ggml.c')
| -rw-r--r-- | ggml/src/ggml.c | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c index 3ff294cc..9e3c4b90 100644 --- a/ggml/src/ggml.c +++ b/ggml/src/ggml.c @@ -1067,7 +1067,11 @@ static const ggml_type_traits_t type_traits[GGML_TYPE_COUNT] = { .from_float = quantize_row_iq2_xxs, .from_float_ref = (ggml_from_float_t)quantize_row_iq2_xxs_ref, .vec_dot = ggml_vec_dot_iq2_xxs_q8_K, +#ifdef __AVX2__ + .vec_dot_type = GGML_TYPE_Q8_2_X4, +#else .vec_dot_type = GGML_TYPE_Q8_K, +#endif .nrows = 1, .row_meta_size = 0, }, |