diff options
| author | Kawrakow <48489457+ikawrakow@users.noreply.github.com> | 2024-03-11 16:53:15 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-03-11 17:53:15 +0200 |
| commit | 44ca159faf4fbe1a7ace13a962845ba7cdfd95ec (patch) | |
| tree | d983c8b474f139f5973d5712886018595dfa2d5d /ggml-common.h | |
| parent | 05b06210c954491cf0f12034b0a62bd4d69ce78b (diff) | |
1.5 bit: we can do even better (#5999)
* iq1_s: we can do even better
Spent one of the 4 scale bits on a signs of a 0.125 shift.
I.e., quants are now -1 + delta, delta, 1 + delta, where delta
is +/- 0.125.
CUDA works, same performance as before.
PPL(LLaMA-v2-7B) is now 11.85!
* iq1_s: make scalar and AVX2 work with the new version
* iq1_s: make Neon work with new version.
~10% drop in performance, so will need some more work.
* iq1_s: make Metal work with new version
* iq1_s: very slightly faster dequantize on Metal
* iq1_s: fix dequantize on the CPU
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml-common.h')
| -rw-r--r-- | ggml-common.h | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/ggml-common.h b/ggml-common.h index 5dd91808..2402b773 100644 --- a/ggml-common.h +++ b/ggml-common.h @@ -645,6 +645,7 @@ GGML_TABLE_BEGIN(uint32_t, iq3s_grid, 512) GGML_TABLE_END() #define NGRID_IQ1S 2048 +#define IQ1S_DELTA 0.125f #if defined(GGML_COMMON_IMPL_C) GGML_TABLE_BEGIN(uint64_t, iq1s_grid, NGRID_IQ1S) 0xffffffffffffffff, 0xffffffffffffff01, 0xffffffffffff0000, 0xffffffffffff01ff, |