diff options
| author | vvhg1 <94630311+vvhg1@users.noreply.github.com> | 2023-10-10 09:31:21 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-10-10 10:31:21 +0300 |
| commit | 11ea5c7d96f2c28e1c99659e08ec0a44574056e2 (patch) | |
| tree | 2ab7b5099157439bce85929ce544ec7a971e3b31 /tests/test-tokenizer-0-falcon.cpp | |
| parent | 95bd60a0a69f57e9a2ff1269667ea484a1a9bb40 (diff) | |
infill. : fix tokenization (#3508)
* infill tokens correction
* serverinfill tokens correction
* removing any leading whitespace from infill suffix and removing leeading space token from suffix when params.escape
* removing any leading whitespace from infill suffix and removing leeading space token from suffix when params.escape
* only rm when params.escape, rm space if possible which is added back or rm added space token
* only rm when params.escape, rm space if possible which is added back or rm added space token
* Revert "only rm when params.escape, rm space if possible which is added back or rm added space token"
This reverts commit 63ba0b621f21077c0e3bc6ba6a327534123cb738.
* fix interactive prompt escaping and fix server infill leading space handling
* rm unnecessary bool check
Diffstat (limited to 'tests/test-tokenizer-0-falcon.cpp')
0 files changed, 0 insertions, 0 deletions