diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2024-10-25 13:08:43 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-10-25 13:08:43 +0200 |
| commit | 6b968f38946117552ffed300771c44ba9b39d3e4 (patch) | |
| tree | dc6b0df69f31ea77d9941d6798a4ef411c688080 /ggml/src/ggml-quants.c | |
| parent | 9114078959b404899fd67e1af45f0dcbee51b47f (diff) | |
Bitnet changes (#106)
* Adapting iq2_bn to work without separate scale tensors
Why? It is becoming burdensome to maintain the special Bitnet
conversion in convert_hf_to_gguf.py, so I thnk it is better
to make iq1_bn and iq2_bn just work with the mainline
conversion script (which does not generate scales).
* Adapting iq1_bn to work without separate scale tensors
* Adapting iq2_bn: CUDA dequantize
* Adapting iq2_bn: CUDA works
* Adapting iq1_bn: CUDA works
* Adapting iq1_bn, iq2_bn: NEON
* Adapting iq1_bn, iq2_bn: Metal
Dequantize works, but there is still something wrong
with the dot products.
* WIP
Absoolutely don't see what is wrong with the iq1_bn and iq2_bn
vector dot product kernels.
* Remove iq1_tn and iq2_tn - Part 1
Now that iq1_bn and iq2_bn have per row scales, there is no
reason to also have iq1_tn and iq2_tn.
* Remove iq1_tn and iq2_tn - Part 2
* Bitnet: use the standard llm_build_kv to build self attention
My main motivation was to enable FA. But FA does not work anyway
because head size is 100 for the Botnet ternary models
(and I had forgotten this little detail).
* Revert "Avoid rebuild of GGML graph for each token (#98)"
This reverts commit f2d315b46f7aacc7df4b86bd8acba387b30e11ca.
As far as I can tell, the commit breaks Metal TG.
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'ggml/src/ggml-quants.c')
| -rw-r--r-- | ggml/src/ggml-quants.c | 2 |
1 files changed, 0 insertions, 2 deletions
diff --git a/ggml/src/ggml-quants.c b/ggml/src/ggml-quants.c index 68ec6126..d18b1981 100644 --- a/ggml/src/ggml-quants.c +++ b/ggml/src/ggml-quants.c @@ -15194,8 +15194,6 @@ bool ggml_validate_row_data(enum ggml_type type, const void * data, size_t nbyte case GGML_TYPE_IQ4_K: break; case GGML_TYPE_IQ5_K: break; case GGML_TYPE_IQ6_K: break; - case GGML_TYPE_IQ2_TN: break; - case GGML_TYPE_IQ1_TN: break; case GGML_TYPE_IQ4_KS: break; case GGML_TYPE_IQ4_KSS: break; case GGML_TYPE_Q4_0_4_4: |