diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 34 |
1 files changed, 13 insertions, 21 deletions
@@ -7,18 +7,10 @@ Inference of [LLaMA](https://arxiv.org/abs/2302.13971) model in pure C/C++ -## ⚠️ TEMPORARY NOTICE ABOUT UPCOMING BREAKING CHANGE ⚠️ - -**The quantization formats will soon be updated: https://github.com/ggerganov/llama.cpp/pull/1305** - -**All `ggml` model files using the old format will not work with the latest `llama.cpp` code after that change is merged** - ---- - **Hot topics:** +- Qauntization formats `Q4` and `Q5` have changed - requantize any old models [(info)](https://github.com/ggerganov/llama.cpp/pull/1405) - [Roadmap May 2023](https://github.com/ggerganov/llama.cpp/discussions/1220) -- [New quantization methods](https://github.com/ggerganov/llama.cpp#quantization) <details> <summary>Table of Contents</summary> @@ -338,18 +330,18 @@ As the models are currently fully loaded into memory, you will need adequate dis Several quantization methods are supported. They differ in the resulting model disk size and inference speed. -| Model | Measure | F16 | Q4_0 | Q4_1 | Q4_2 | Q5_0 | Q5_1 | Q8_0 | -|------:|--------------|-------:|-------:|-------:|-------:|-------:|-------:|-------:| -| 7B | perplexity | 5.9066 | 6.1620 | 6.0910 | 6.1466 | 5.9862 | 5.9481 | 5.9069 | -| 7B | file size | 13.0G | 4.0G | 4.8G | 4.0G | 4.4G | 4.8G | 7.1G | -| 7B | ms/tok @ 4th | 128 | 56 | 61 | 84 | 91 | 95 | 75 | -| 7B | ms/tok @ 8th | 128 | 47 | 55 | 48 | 53 | 59 | 75 | -| 7B | bits/weight | 16.0 | 5.0 | 6.0 | 5.0 | 5.5 | 6.0 | 9.0 | -| 13B | perplexity | 5.2543 | 5.3863 | 5.3607 | 5.3513 | 5.2856 | 5.2706 | 5.2548 | -| 13B | file size | 25.0G | 7.6G | 9.1G | 7.6G | 8.4G | 9.1G | 14G | -| 13B | ms/tok @ 4th | 239 | 104 | 113 | 160 | 176 | 185 | 141 | -| 13B | ms/tok @ 8th | 240 | 85 | 99 | 97 | 108 | 117 | 147 | -| 13B | bits/weight | 16.0 | 5.0 | 6.0 | 5.0 | 5.5 | 6.0 | 9.0 | +| Model | Measure | F16 | Q4_0 | Q4_1 | Q5_0 | Q5_1 | Q8_0 | +|------:|--------------|-------:|-------:|-------:|-------:|-------:|-------:| +| 7B | perplexity | 5.9066 | 6.1620 | 6.0910 | 5.9862 | 5.9481 | 5.9069 | +| 7B | file size | 13.0G | 4.0G | 4.8G | 4.4G | 4.8G | 7.1G | +| 7B | ms/tok @ 4th | 128 | 50 | 54 | 75 | 83 | 75 | +| 7B | ms/tok @ 8th | 123 | 44 | 52 | 53 | 58 | 72 | +| 7B | bits/weight | 16.0 | 5.0 | 6.0 | 5.5 | 6.0 | 9.0 | +| 13B | perplexity | 5.2543 | 5.3863 | 5.3607 | 5.2856 | 5.2706 | 5.2548 | +| 13B | file size | 25.0G | 7.6G | 9.1G | 8.4G | 9.1G | 14G | +| 13B | ms/tok @ 4th | 239 | 93 | 101 | 150 | 164 | 141 | +| 13B | ms/tok @ 8th | 240 | 81 | 96 | 96 | 104 | 136 | +| 13B | bits/weight | 16.0 | 5.0 | 6.0 | 5.5 | 6.0 | 9.0 | ### Perplexity (measuring model quality) |