diff options
| author | Galunid <karolek1231456@gmail.com> | 2023-10-24 09:17:17 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-10-24 09:17:17 +0200 |
| commit | daab3d7f45832e10773c99f3484b0d5b14d86c0c (patch) | |
| tree | 432092c5aec7c775ab6e33b968564bd0a1e4a187 /examples/batched/batched.cpp | |
| parent | 469c9addef75893e6be12edda852d12e840bf064 (diff) | |
Add more tokenizer tests (#3742)
* Add more tokenizer tests
* Add starcoder
* Update test vocab files
* Restrict bpe tokenizer tests to unicode planes
* Update comment
* Comment cosmetics
* Remove bloom vocab/test
Diffstat (limited to 'examples/batched/batched.cpp')
0 files changed, 0 insertions, 0 deletions