diff options
| author | Georgi Gerganov <ggerganov@gmail.com> | 2023-10-27 17:01:23 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-10-27 17:01:23 +0300 |
| commit | 2f9ec7e271220a78fe27c9e6ccbcc0dda31cda0f (patch) | |
| tree | e36dcd292a133c5106376c6ddd90c68f446ddae6 /examples/server/api_like_OAI.py | |
| parent | 34b2a5e1ee4fe6295fb4420eb91131d743694c65 (diff) | |
cuda : improve text-generation and batched decoding performance (#3776)
* cuda : prints wip
* cuda : new cublas gemm branch for multi-batch quantized src0
* cuda : add F32 sgemm branch
* cuda : fine-tune >= VOLTA params + use MMQ only for small batches
* cuda : remove duplicated cuBLAS GEMM code
* cuda : add CUDA_USE_TENSOR_CORES and GGML_CUDA_FORCE_MMQ macros
* build : add compile option to force use of MMQ kernels
Diffstat (limited to 'examples/server/api_like_OAI.py')
0 files changed, 0 insertions, 0 deletions