diff options
| author | Georgi Gerganov <ggerganov@gmail.com> | 2024-05-04 08:32:32 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-04 08:32:32 +0300 |
| commit | 92139b90af4841d7fd060b526bdd443b621770ff (patch) | |
| tree | 9679c3de1b39970ca73b5bd988c63ddac0359ca6 /models/ggml-vocab-deepseek-coder.gguf.inp | |
| parent | a2ac89d6efb41b535778bfeaecaae8fe295b6ed3 (diff) | |
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
* tests : add test-tokenizer-0.sh
* unicode : add all unicode number ranges
* starcoder : fix pre-tokenizer
* tests : add test that fails with DeepSeek tokenizers
* falcon : fix regex
* unicode : regenerate unicode tables
* refact : add tokenizer model
* lint : fix
* tests : disable failing tests
ggml-ci
* refact : add tests files
ggml-ci
* convert : print -> logging
ggml-ci
* lint : fix
* unicode : digit -> number
* phi-3 : update
Diffstat (limited to 'models/ggml-vocab-deepseek-coder.gguf.inp')
| -rw-r--r-- | models/ggml-vocab-deepseek-coder.gguf.inp | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/models/ggml-vocab-deepseek-coder.gguf.inp b/models/ggml-vocab-deepseek-coder.gguf.inp index 0389f00c..0a89107c 100644 --- a/models/ggml-vocab-deepseek-coder.gguf.inp +++ b/models/ggml-vocab-deepseek-coder.gguf.inp @@ -1,3 +1,7 @@ +ied 4 ½ months +__ggml_vocab_test__ +Führer +__ggml_vocab_test__ __ggml_vocab_test__ |