diff options
| author | Andrew Godfrey <AndrewGodfrey@users.noreply.github.com> | 2023-11-17 02:23:11 -0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-11-17 11:23:11 +0100 |
| commit | 947f64f1630bb8b0b363a3bb5e29e11425312d57 (patch) | |
| tree | 9d8e8ecea7cdf6bd3b6cb192900f82dec82fb825 /llama.cpp | |
| parent | b83e149ec6264d078e6a47412e7347bf5c2bfcc9 (diff) | |
finetune : zero the loraB initial vectors (#4082)
* finetune : zero the loraB initial vectors
Without this, the first iteration is starting out far from the base model, instead of exactly on it.
Zeroing loraB is what the paper recommends. loralib also zeroes at least one of the init vector pairs
(though it departs from the paper in using a different distribution for the other vector, in some cases).
* tabs to spaces
* Use ggml_set_zero instead of adding a new function
Diffstat (limited to 'llama.cpp')
0 files changed, 0 insertions, 0 deletions