diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2025-03-27 05:49:16 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-03-27 05:49:16 +0100 |
| commit | d0b52076da0261f291b01f1ffa44884c8b2cdb1c (patch) | |
| tree | 93abea8ae30140fbd6733af91eede57c2243e91d /ggml/src/iqk/iqk_quantize.h | |
| parent | a22250df93fd833a6cb7f310b159ad1b54e4d582 (diff) | |
Use bf16 instead of fp16 block scales for q8_1 (#292)
* WIP - not working
* q8_0 without bells and wistles works
* It works for q8_0
* Use bf16 instead of f16,int16
* q4_0_r8
* q5_0_r4
* q6_0_r4
* Also q4_1 and q5_1
* q8_0_r8 on avx2
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/iqk/iqk_quantize.h')
| -rw-r--r-- | ggml/src/iqk/iqk_quantize.h | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/ggml/src/iqk/iqk_quantize.h b/ggml/src/iqk/iqk_quantize.h index dd148f2e..478bd0de 100644 --- a/ggml/src/iqk/iqk_quantize.h +++ b/ggml/src/iqk/iqk_quantize.h @@ -238,6 +238,7 @@ void quantize_row_q8_K32(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, void quantize_row_q8_KR8(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k); void quantize_row_q8_0_x4(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k); void quantize_row_q8_1_x4(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k); +void quantize_row_q8_2_x4(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k); void repack_f32_bf16_r16 (const void * GGML_RESTRICT src, void * GGML_RESTRICT dst, int64_t nrows, int64_t n_per_row); void repack_bf16_bf16_r16(const void * GGML_RESTRICT src, void * GGML_RESTRICT dst, int64_t nrows, int64_t n_per_row); |