summaryrefslogtreecommitdiff
path: root/tests/test-tokenizer-1-bpe.cpp
AgeCommit message (Expand)Author
2024-03-11llama : refactor unicode stuff (#5992)Georgi Gerganov
2024-02-16ggml : add numa options (#5377)bmwl
2024-02-13tests : multi-thread the tokenizer tests (#5474)Georgi Gerganov
2023-10-24Add more tokenizer tests (#3742)Galunid
2023-10-03Work on the BPE tokenizer (#3252)goerch