diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2025-06-17 07:12:48 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-06-17 07:12:48 +0300 |
| commit | 0f8f8b32e2d0c7e3ac8bbafee6965dcd1305d002 (patch) | |
| tree | debb40f79053c891ff1cfa9839a7cca8505c1a04 /ggml/src/ggml-common.h | |
| parent | 6fc5bbb657525bb1ef20b682e1cc4ab5fd44aba6 (diff) | |
Much faster CPU prompt processing (part 1) (#531)
* q6_K dequantizing GEMM
* Much easier: just use different vec_dot types!
* WIP
* Finally q6_K x q8_2_x4 dot product works
* Very slightly better
* We don't need the changes in ggml.c
* Fix AVX2
* iq2_xs
* Fix AVX2
* iq2_s
* q3_K
* Fix q8_k_r8 on Zen4
* q3_K: repack to q8_k_r8 instead of q8_0_r8
With that we hit 360 t/s for LlaMA-3.1-8B on a Ryzen-7950X.
q8_k_r8 is 386 t/s, so for a batch size of 512 repacking costs
~7% of the time taken by the actual GEMM.
* q3_K: don't scale when all quants in a block are <= 127 when repacking
* iq2_s: repack to q8_k_r8 instead of q8_0_r8
* iq2_xs: rapck to q8_k_r8
* WIP
* iq2_xs: repack to q8_k_r8
* iq3_xxs: repack to q8_k_r8
* iq3_s: use q8_k_r8
* iq1_s: repack to q8_k_r8
* iq1_m: repack to q8_k_r8
* iq1_m: slightly faster
* Slightly faster
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/ggml-common.h')
| -rw-r--r-- | ggml/src/ggml-common.h | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/ggml/src/ggml-common.h b/ggml/src/ggml-common.h index 5fe27b29..2bfe5d39 100644 --- a/ggml/src/ggml-common.h +++ b/ggml/src/ggml-common.h @@ -386,10 +386,11 @@ static_assert(sizeof(block_q6_k_r4) == 4*sizeof(ggml_half) + QK_K/4 + 3*QK_K, "w // This is only used for intermediate quantization and dot products typedef struct { float d; // delta + float sum; // sum of quants in the entire block int8_t qs[QK_K]; // quants int16_t bsums[QK_K/16]; // sum of quants in groups of 16 } block_q8_K; -static_assert(sizeof(block_q8_K) == sizeof(float) + QK_K + QK_K/16*sizeof(int16_t), "wrong q8_K block size/padding"); +static_assert(sizeof(block_q8_K) == 2*sizeof(float) + QK_K + QK_K/16*sizeof(int16_t), "wrong q8_K block size/padding"); typedef struct { float d; // delta int8_t qs[64]; // quants |