diff options
| author | Kawrakow <48489457+ikawrakow@users.noreply.github.com> | 2024-01-22 12:43:33 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-01-22 12:43:33 +0200 |
| commit | 66d575c45c5a370d668f9c3283cdf348e2329fa2 (patch) | |
| tree | 035e052b116f301508225f897f1943e6eb1b3e19 /examples/imatrix/imatrix.cpp | |
| parent | 57744932c64266359ee905518de7e096c0295d8c (diff) | |
llama : add Q3_K_XS (#5060)
* Add Q3_K_XS - intermediate size between Q2_K and Q3_K_S
* Q3_K_XS: quanize first 1/8 of ffn_down layers with Q4_K
Together with an importance matrix, this brings perplexity
for LLaMA-v2-70B below the perplexity of the former Q2_K
with a 800 MB smaller quantized model size.
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'examples/imatrix/imatrix.cpp')
0 files changed, 0 insertions, 0 deletions