summaryrefslogtreecommitdiff
path: root/tests/test-tokenizer-random.py
AgeCommit message (Expand)Author
2024-07-27Merge mainline llama.cpp (#3)Kawrakow
2024-06-18tokenizer : BPE fixes (#7530)jaime-m-p
2024-06-04Per token attributes (#7685)jaime-m-p
2024-05-28Tokenizer WPM fixes (#7500)jaime-m-p
2024-05-21Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)jaime-m-p
2024-05-20Tokenizer SPM fixes for phi-3 and llama-spm (#7375)jaime-m-p
2024-05-18Unicode codepoint flags for custom regexs (#7245)jaime-m-p
2024-05-09llama3 custom regex split (#6965)jaime-m-p