diff options
| author | jaime-m-p <167997752+jaime-m-p@users.noreply.github.com> | 2024-05-21 14:39:48 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-21 14:39:48 +0200 |
| commit | d7e852c1bc8e85bf62a6f1aede08cd2de723404a (patch) | |
| tree | 46323a83d73f66727459aee88a995e946a78e005 /common/sampling.cpp | |
| parent | 917dc8cfa67a72fb7c8bf7392270da3bf4833af4 (diff) | |
Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)
* Update brute force test: add_special
* Update brute force test: default values for add_bos_token and add_eos_token
* Enable rtrim when pre-inserting BOS
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Revert "server : fix test regexes"
Diffstat (limited to 'common/sampling.cpp')
0 files changed, 0 insertions, 0 deletions