diff options
| author | John <78893154+cmp-nct@users.noreply.github.com> | 2024-01-10 14:37:09 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-01-10 15:37:09 +0200 |
| commit | d34633d8db6c2e400355de4862cd699154ecc73f (patch) | |
| tree | 9047425d7bb97e369437a2bd28fa0405e27a5533 /examples/server | |
| parent | 4f56458d34cb13dcbf69aca650e9bf77d5497e6f (diff) | |
clip : support more quantization types (#4846)
Uses ggml functions instead of hardcoded names and adds support to quantize into the modern Q-K variants.
This is just the bare minimum to get k-types working - a more refined choice of types would be needed to get best quality on low quantizations.
I ran a few tests, it doesn't break anything I could notice and a Q6_K ViT works almost as well as Q8_0 but 3 times the inference speed.
Diffstat (limited to 'examples/server')
0 files changed, 0 insertions, 0 deletions