diff options
| author | Kawrakow <48489457+ikawrakow@users.noreply.github.com> | 2023-08-27 16:50:33 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-08-27 16:50:33 +0300 |
| commit | 463173a6c0ff353055eb90665794884c888c790f (patch) | |
| tree | 4868e5ed0a6924410c91b149a6a630ea75ea06de /examples/embedding/embedding.cpp | |
| parent | eaa13a48ff4136f01c1cdb79cacd61b67ec53095 (diff) | |
llama : speedup tokenization (#2831)
* Speedup tokenization
On current master it takes ~3.2 seconds to tokenize
Wikitext. With this change it becomes ~525 ms.
* Fixit: it was missing the piece after the last found occurence
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Diffstat (limited to 'examples/embedding/embedding.cpp')
0 files changed, 0 insertions, 0 deletions