summaryrefslogtreecommitdiff
path: root/examples/tokenize/tokenize.cpp
AgeCommit message (Collapse)Author
2023-11-18tokenize example: Respect normal add BOS token behavior (#4126)Kerfuffle
Allow building with Makefile
2023-11-17tokenize : fix trailing whitespaceGeorgi Gerganov
2023-11-17examples : add tokenize (#4039)zakkor