diff options
| author | Georgi Gerganov <ggerganov@gmail.com> | 2023-09-05 08:46:17 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-09-05 08:46:17 +0300 |
| commit | 921772104ba2219bfdc2b2980d05ebc0aa0c92a4 (patch) | |
| tree | c9ba8f0f8d0471c9c3c85a6963b322affa0f4f97 /grammars | |
| parent | 2ba85c8609309a59d49c45ab43c31800b7ba141c (diff) | |
speculative : add grammar support (#2991)
* speculative : add grammar support
* grammars : add json_arr.gbnf
* grammar : add comments to new grammar file
* grammar : remove one nested level
* common : warm-up with 2 tokens - seems to work better
* speculative : print draft token pieces
* speculative : reuse grammar parser + better logs and comments
* speculative : avoid grammar_mem
* make : fix speculative build
Diffstat (limited to 'grammars')
| -rw-r--r-- | grammars/json_arr.gbnf | 34 |
1 files changed, 34 insertions, 0 deletions
diff --git a/grammars/json_arr.gbnf b/grammars/json_arr.gbnf new file mode 100644 index 00000000..ef53e77a --- /dev/null +++ b/grammars/json_arr.gbnf @@ -0,0 +1,34 @@ +# This is the same as json.gbnf but we restrict whitespaces at the end of the root array +# Useful for generating JSON arrays + +root ::= arr +value ::= object | array | string | number | ("true" | "false" | "null") ws + +arr ::= + "[\n" ws ( + value + (",\n" ws value)* + )? "]" + +object ::= + "{" ws ( + string ":" ws value + ("," ws string ":" ws value)* + )? "}" ws + +array ::= + "[" ws ( + value + ("," ws value)* + )? "]" ws + +string ::= + "\"" ( + [^"\\] | + "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes + )* "\"" ws + +number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws + +# Optional space: by convention, applied in this grammar after literal chars when allowed +ws ::= ([ \t\n] ws)? |