diff options
| author | Galunid <karolek1231456@gmail.com> | 2023-10-23 21:46:00 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-10-23 21:46:00 +0200 |
| commit | 69a6735087c3634963c642fd69f0851ac479cd78 (patch) | |
| tree | b4a0a0037e1d748e1dbf531ab5c83da5fcd7bd2e /examples/infill/infill.cpp | |
| parent | 5be6c803fa5378f62a1590f3ad8c6b64c7c0c2ce (diff) | |
Update special token handling in conversion scripts for gpt2 derived tokenizers (#3746)
We still have the heads up in `README.md` regarding `bpe` tokenizers and this patch is needed for
- a couple of tokenizer tests
- some more `special` and `non-special` added tokens handling (as far as I understand it)
* Update special token handling
* Add mpt
Diffstat (limited to 'examples/infill/infill.cpp')
0 files changed, 0 insertions, 0 deletions