diff options
| author | Evan Jones <evan.q.jones@gmail.com> | 2023-07-23 23:58:10 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-07-23 23:58:10 -0400 |
| commit | 84e09a7d8bc4ab6d658b5cd81295ac0add60be78 (patch) | |
| tree | 934c5480d917325ac8baa29f4edfae99137b56bb /grammars/arithmetic.gbnf | |
| parent | 2f9cf974a066ac0e03fbb235d834b01b0164d743 (diff) | |
llama : add grammar-based sampling (#1773)
* llama, main : constrain sampling to grammar
* allow loading grammar from file
* fix whitespace errors
* handle & print parser errors
* add comments to grammar syntax and allow newlines where unambiguous
* add missing include
* support alternates in root rule
* fix bugs with empty token and EOS
* adjust JSON grammar
* remove swp file
* rewrite ternary expressions
Co-authored-by: Henri Vasserman <henv@hot.ee>
* use struct for grammar elements and add Unicode support
* add unicode escapes
* add inverse char ranges
* only sample full tokens (no peeking or truncation)
* llama : minor style changes
blindly applied in online editor - hopefully I didn't break something
* update help text
* add warning message if EOS is disabled
---------
Co-authored-by: Henri Vasserman <henv@hot.ee>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'grammars/arithmetic.gbnf')
| -rw-r--r-- | grammars/arithmetic.gbnf | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/grammars/arithmetic.gbnf b/grammars/arithmetic.gbnf new file mode 100644 index 00000000..3aa95a9d --- /dev/null +++ b/grammars/arithmetic.gbnf @@ -0,0 +1,6 @@ +root ::= (expr "=" ws term "\n")+ +expr ::= term ([-+*/] term)* +term ::= ident | num | "(" ws expr ")" ws +ident ::= [a-z] [a-z0-9_]* ws +num ::= [0-9]+ ws +ws ::= [ \t\n]* |