diff options
| author | Kawrakow <iwankawrakow@gmail.com> | 2024-10-21 12:16:54 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-10-21 12:16:54 +0200 |
| commit | dbf951df1594a3dec36eca9ab81a0f7ba81b11cd (patch) | |
| tree | a1692718a4602927fe34dc313841aec582e7fe58 /examples | |
| parent | f2d315b46f7aacc7df4b86bd8acba387b30e11ca (diff) | |
Enable IQ4_NL for KV-cache in token generation using Flash Attention (#99)
* Enable IQ4_NL for V-cache in token generation
* We don't need these
* Update printour of allowed quantized KV-cache combinations
* Add IQ4_NL + IQ4_NL to FA
This is a better alternative than Q4_0 + Q4_0 for the VRAM poor.
* Remove file added by mistake
* Fix typo, which is not really a bug
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'examples')
0 files changed, 0 insertions, 0 deletions