diff options
| author | Aarni Koskela <akx@iki.fi> | 2024-02-13 18:18:16 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-02-13 18:18:16 +0200 |
| commit | c4e6dd59e45ef7b14f7763fb073b517395dc176c (patch) | |
| tree | 2cbd0de3c717bd1cc0fdf6f877035dcce6194d5b /gguf-py | |
| parent | 037259be689353081e7bae3c1ab4ab18e7fbe8c9 (diff) | |
llama : allow raw byte in SPM vocabs; don't crash on nl 404 (#5478)
* common : don't crash if newline token is not found
* common : llama_byte_to_token: allow falling back to finding just the token byte in SPM vocabs
Diffstat (limited to 'gguf-py')
0 files changed, 0 insertions, 0 deletions