 |
ML Newsnew | past | best | rss | submit | about |
| Extending Context Window of Large Language Models via Positional Interpolation (arxiv.org) | |
| 10 points by tosh on Jun 28, 2023 | hide | past | pdf | 1 comment on HN | |
|
In plain words: Instead of pushing word position numbers past the range a model saw in training, which can wreck how it weighs words, this squeezes them to fit the trained range. After under 1000 fine-tuning steps, it handles 32,768-word contexts and still answers short-context tasks well. AbstractWe present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure. | |
| add comment on HN |
This is a pretty wild result. Almost all of the open source models folks have been using from HF etc can suddenly easily be extended to have an enormous context window for little to no effort. You can already see these models being distributed in the most popular formats for inference, like GGML, on The Bloke's HuggingFace profile!