diff options
| author | Kawrakow <48489457+ikawrakow@users.noreply.github.com> | 2024-09-09 14:56:34 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-09-09 14:56:34 +0300 |
| commit | 8c86231f9306c81dc291c4c4a16f88bbc7c97793 (patch) | |
| tree | d49325de2775076e1f71ddf94667d0cd02db3cc5 /ggml/src/ggml-quants.c | |
| parent | bf4b19b474b78a6ddfa1f0fe19f76f3c7ac92030 (diff) | |
Adding IQ1_TN - 1.6875 bpw for TriLM ternary models (#44)
* Adding iq1_tn - 1.6875 bpw for TriLM ternary models
* iq1_tn: NEON
* iq1_tn: faster NEON
* iq2_bn: improve performance on NEON
We now get TG-128 = 100 t/s for Bitnet-3B-1.58b!
* iq1_tn: improve AVX2
PP-512 goes to 533 t/s up from 455.
TG-128 @ 2 threads goes to 16.6 t/s up from 14.2.
However, we seem to have a bottleneck somewhere as
TG saturates at 8 threads.
* iq1_tn: improve Zen4
PP-512 goes to 485 t/s up from 352. With FA we get 545 t/s up from 380.
TG-128 @ 1 thread goes to 12.4 t/s up from 10.4.
However, we seem to have a bottleneck somewhere as
TG saturates at 8 threads.
* iq2_bn: improve on Zen4
We now get PP-512 = 614 t/s up from 542 t/s
* iq2_bn: improve AVX2 implementation
We now get PP-512 = 753 t/s up from 680 t/s.
* Remove unnecessary barrier in ggml_compute_forward_mul_mat
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/ggml-quants.c')
| -rw-r--r-- | ggml/src/ggml-quants.c | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/ggml/src/ggml-quants.c b/ggml/src/ggml-quants.c index 981fb54b..a9a25761 100644 --- a/ggml/src/ggml-quants.c +++ b/ggml/src/ggml-quants.c @@ -15015,6 +15015,7 @@ bool ggml_validate_row_data(enum ggml_type type, const void * data, size_t nbyte case GGML_TYPE_IQ5_K: break; case GGML_TYPE_IQ6_K: break; case GGML_TYPE_IQ2_TN: break; + case GGML_TYPE_IQ1_TN: break; case GGML_TYPE_Q4_0_4_4: case GGML_TYPE_Q4_0_4_8: { |