diff options
| author | jaime-m-p <167997752+jaime-m-p@users.noreply.github.com> | 2024-05-20 20:15:57 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-20 20:15:57 +0200 |
| commit | 917dc8cfa67a72fb7c8bf7392270da3bf4833af4 (patch) | |
| tree | 170c39ba79ae72036105970d92548da420699ba1 /common/common.h | |
| parent | fabf30b4c4fca32e116009527180c252919ca922 (diff) | |
Tokenizer SPM fixes for phi-3 and llama-spm (#7375)
* Update brute force test: special tokens
* Fix added tokens
- Try to read 'added_tokens.json'.
- Try to read 'tokenizer_config.json'.
- Try to read 'tokenizer.json'.
* Fix special tokens rtrim
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* server : fix test regexes
Diffstat (limited to 'common/common.h')
0 files changed, 0 insertions, 0 deletions