diff options
| author | Haoxiang Fei <tonyfettes@tonyfettes.com> | 2024-05-11 16:12:06 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-11 11:12:06 +0300 |
| commit | f99e1e456eaf69cc38c1982a2693ce41c0f897ef (patch) | |
| tree | f6bb7dd98afdc852fa428c77c53bf8e72fb69b5e /examples/server/server.cpp | |
| parent | 5ae3426b0b64672991563d4c28b2018b9f961467 (diff) | |
llama : lookup word in vocab before doing BPE merges (#7193)
* fix: llama-3 ignore_merges
* test: add test for llama-3 bpe ignore_merges
* fix: set ignore_merges only for llama-3
* fix: test-tokenizer-1-bpe --ingore-merges detection
* fix: copy to fix fallthrough
* fix: change ignore_merges to bool
* fix: add ignore merges tests to cmake
* llama : alternative merge ignore logic
---------
Co-authored-by: Haoxiang Fei <feihaoxiang@idea.edu.cn>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'examples/server/server.cpp')
0 files changed, 0 insertions, 0 deletions