diff options
| author | FK <sozforex@gmail.com> | 2023-09-13 08:50:46 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-09-13 08:50:46 +0200 |
| commit | 84e723653ca99d51a74b454984acf2c077468561 (patch) | |
| tree | 62ddb7a849eb2ecf10dc831bf4ea960320e4dd5f /examples/speculative/speculative.cpp | |
| parent | b52b29ab9d601bb298050bcd2261169bc917ba2c (diff) | |
speculative: add --n-gpu-layers-draft option (#3063)
Diffstat (limited to 'examples/speculative/speculative.cpp')
| -rw-r--r-- | examples/speculative/speculative.cpp | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/examples/speculative/speculative.cpp b/examples/speculative/speculative.cpp index 822d7b52..2cd153f9 100644 --- a/examples/speculative/speculative.cpp +++ b/examples/speculative/speculative.cpp @@ -42,6 +42,7 @@ int main(int argc, char ** argv) { // load the draft model params.model = params.model_draft; + params.n_gpu_layers = params.n_gpu_layers_draft; std::tie(model_dft, ctx_dft) = llama_init_from_gpt_params(params); // tokenize the prompt |