diff options
| author | Kawrakow <48489457+ikawrakow@users.noreply.github.com> | 2024-08-07 07:56:09 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-08-07 07:56:09 +0200 |
| commit | a9f302ebe2373321c12b01d8760904901aa064a4 (patch) | |
| tree | 7953bbff2ebd6bf9130cea52d17995aea3cd65d5 /ggml/src/ggml-common.h | |
| parent | b409c153636d27473970abd3a9c9400b6287d400 (diff) | |
Adding IQ2_TN for use with ternary models (#13)
* iq2_tn: TriLM specific 2.0625 bpw quantization
Quantize/dequantize/scale dot product.
I get 46 t/s for the TriLM-3.9B with any SIMD!
Finally a compiler doing a decent job auto-vectorizing the
scalar implementation.
* iq2_tn: AVX512
Just reusing the k-quants template gets us to PP-512 = 376 t/s,
TG-128 = 47.6 t/s for TriLM-3.9B.
* iq2_tn: AVX512
With this tweak we get to PP-512 = 431 t/s.
* iq2_tn: AVX512
With this tweak we get TG-128 = 19.58 / 35.18 t/s for 1 / 2 threads.
At 4 threads we saturate at 48.41 t/s, and then performance slowly
degrades with increasing number of threads.
* iq2_tn: AVX2
PP512 = 440 t/s on the Ryzen-5975WX.
We should be able to do better.
* iq2_tn: initial NEON version
* iq2_tn: NEON
For TriLM-3.9B running on the M2-Max we get PP-512 = 193.5 t/s,
TG-128 = 75.5 t/s. This is in line with what we have for
iq2_bn ant 3.3B Bitnet.
* iq2_tn: Metal
For TriLM-3.9B on a 30-core M2-Max we get PP-512 = 890 t/s,
TG-128 = 98.5 t/s.
* iq2_tn: CUDA
For TriLM-3.9B running on RTX-4080 we get PP-512 = 9936 t/s,
TG-128 = 299.2 t/s.
* iq2_tn: AVX2 PP improvement
We now get PP-512 = 490.73 t/s for TriLM-3.9B on the Ryzen-5975WX.
We have PP-512 = 636.61 t/s for Bintnet-3B quantized with iq2_bn.
Bintnet-3B is actually 3.4B, TriLM-3.9B is 3.99B, so we would
expect 3.43/3.99 * 636 = 546 t/s, so it seems we still have something
that is not quite optimal in iq2_tn.
* iq2_tn: small NEON improvement
For TriLM-3.9B we now get PP-512 = 206.6 t/s and TG-128 = 76.4 t/s.
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/ggml-common.h')
| -rw-r--r-- | ggml/src/ggml-common.h | 12 |
1 files changed, 10 insertions, 2 deletions
diff --git a/ggml/src/ggml-common.h b/ggml/src/ggml-common.h index 423797b6..5847d903 100644 --- a/ggml/src/ggml-common.h +++ b/ggml/src/ggml-common.h @@ -407,7 +407,7 @@ typedef struct { static_assert(sizeof(block_iq1_m) == QK_K/8 + QK_K/16 + QK_K/32, "wrong iq1_m block size/padding"); // -// Bitnet - implemented as 1.75 bpw +// Bitnet - implemented as 1.625 bpw // The block scale is a waste, but it allows us to plug it in without any additional // changes to ggml. // @@ -418,13 +418,21 @@ typedef struct { } block_iq1_bn; static_assert(sizeof(block_iq1_bn) == 13, "wrong iq1_bn block size/padding"); // -// Bitnet - implemented as 2.25 bpw +// Bitnet - implemented as 2.0 bpw // #define QK_IQ2BN 64 typedef struct { uint8_t qs[QK_IQ2BN/4]; } block_iq2_bn; static_assert(sizeof(block_iq2_bn) == QK_IQ2BN/4, "wrong iq2_bn block size/padding"); +// +// TriLM - implemented as 2.0625 bpw +// +typedef struct { + ggml_half d; + uint8_t qs[QK_K/4]; +} block_iq2_tn; +static_assert(sizeof(block_iq2_tn) == sizeof(ggml_half) + QK_K/4, "wrong iqt_bn block size/padding"); // Used by IQ1_M quants typedef union { |