diff options
| author | DAN™ <dranger003@gmail.com> | 2024-05-05 01:19:30 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-05 08:19:30 +0300 |
| commit | 889bdd76866ea31a7625ec2dcea63ff469f3e981 (patch) | |
| tree | bcfaea83b66f89181980b64dba57a8c8f8f31a49 /tests/test-tokenizer-0.sh | |
| parent | 6fbd43221167bf96112f899daf22c127b282cbcf (diff) | |
command-r : add BPE pre-tokenization (#7063)
* Add BPE pre-tokenization for Command-R/R+.
* Bump transformers convert requirement.
* command-r : add individual digits regex
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'tests/test-tokenizer-0.sh')
0 files changed, 0 insertions, 0 deletions