diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2025-06-11 11:12:30 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-06-11 11:12:30 +0300 |
| commit | e56061fa12e5fcb7a9a8fa5fca9e74c20d166bcc (patch) | |
| tree | f0627d22fac777db7e397119b2f38421a2b289b8 /ggml/src/iqk/iqk_gemm_iquants.h | |
| parent | 3c1f2c68fdbb9e5be26aad85902c3d86057d69b5 (diff) | |
IQ2_XXS: much faster CPU prompt processing (#515)
* Much faster iq2_xxs GEMM
PP-512 = 290 t/s vs ~110 t/s (iq2_xxs) or 148 t/s (iq2_xxs_r4) on main.
* iq2_xxs: q8_2_x4 GEMM
* iq2_xxs: use template for q8_2_x4 GEMM
* Fix AVX2
* Cleanup
* NEON is not working yet, so still use Q8_K GEMM
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/iqk/iqk_gemm_iquants.h')
| -rw-r--r-- | ggml/src/iqk/iqk_gemm_iquants.h | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/ggml/src/iqk/iqk_gemm_iquants.h b/ggml/src/iqk/iqk_gemm_iquants.h index 4182526a..82d1bb80 100644 --- a/ggml/src/iqk/iqk_gemm_iquants.h +++ b/ggml/src/iqk/iqk_gemm_iquants.h @@ -8,4 +8,6 @@ bool iqk_set_kernels_iquants(int ne00, int typeA, int typeB, std::array<mul_mat_t, IQK_MAX_NY>& kernels, mul_mat_t& func16); +bool iqk_convert_iquants_q80_r8(int type, int n, const void * vx, size_t bx, void * vy, int nrc_x); + #endif |