| 31. |
Does Pretraining Teach Chemical Language Models Chemistry? (arxiv.org) |
| They tested whether pretraining on molecules teaches chemistry by comparing pretrained chemical language models across 64 property-prediction tasks with classical-feature baselines. More pretraining brought no gains and stayed below those baselines; only pretraining on physicochemical descriptors helped, and some was worse than starting untrained. |
|
3 points by yz-exodao 13 days ago | hide | past | pdf | discuss
|
| 32. |
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval (arxiv.org) |
| They tested long-context retrieval by adding harmless filler or statements that give the same key a rival value. Filler cost nothing, but one rival cut accuracy to 0.16 to 0.33, and how much attention stayed on the evidence predicted the failure far better than length. |
|
3 points by liusci 13 days ago | hide | past | pdf | discuss
|
| 33. |
You don't need long context (arxiv.org) |
| Because later words cost up to seven times more to generate, the approach repeatedly squeezes the conversation into a short summary instead of keeping everything. Accuracy barely dropped on long tasks, even when the model took just one action per summary. |
|
3 points by craigmart 13 days ago | hide | past | pdf | discuss
|
| 34. |
A Fix for Softmax and Cross-Entropy (arxiv.org) |
| Softmax and cross-entropy code has rounding errors that erase a needed gradient or invent one when it should be zero; rewriting the math fixes both. The fix removes training jumps and accuracy crashes, and shows output values stop growing once weight decay balances Adam's epsilon. |
|
4 points by theanonymousone 20 days ago | hide | past | pdf | discuss
|
| 35. |
Automatic Construction of Strong Simple Baselines (arxiv.org) |
| A language model builds simple baselines for 36 deep-learning tasks from top science journals, to check whether new methods really help. They recover a median 95.5% of the gap between a trivial guess and the deep model, matching or beating it in 12 studies. |
|
3 points by cellington 14 days ago | hide | past | pdf | discuss
|
| 36. |
GSLM2 MultiStream Speech Rep Learning and Gen Spoken Language Modeling (arxiv.org) |
| Instead of squeezing speech into a single stream of word-like units, this system learns several layers of units capturing different speech details and predicts them together. It cuts the gap to text-trained language models by about 100-fold on grammar and word tests. |
|
3 points by mrkn1 14 days ago | hide | past | pdf | discuss
|
| 37. |
FreeToken: Datacenter-class intelligence on the hardware you own (arxiv.org) |
| FreeToken keeps a huge model's many specialist parts moving between memory, disk, and the GPU as space frees up, instead of sticking to one fixed offloading plan. That lets a single workstation GPU run a 753B-parameter model that normally needs a datacenter. |
|
3 points by lioeters 15 days ago | hide | past | pdf | 3 comments
|
| 38. |
Context as Infrastructure: Knowledge Compilation for Retrieval in Agent Harness (arxiv.org) |
| An agent that writes its own searches answers questions well but wastes its window on repeated queries and unused passages. Pre-compiling documents into a tree of claim-and-evidence notes cut the text it processes by 91% when the evidence was already there, with no accuracy change. |
|
3 points by yingfeng 15 days ago | hide | past | pdf | discuss
|
| 39. |
UlaCache: Unified Logical Addressing for Sparse LLM Serving (arxiv.org) |
| UlaCache gives each cached token a fixed ID pointing to its spot on GPU or CPU, so the model keeps what it needs regardless of where it sits. With equal memory budgets, it decoded up to 2.51 times faster than the best prior system. |
|
3 points by wslh 17 days ago | hide | past | pdf | discuss
|
| 40. |
Collaborative AI agents for autonomous experimental research (arxiv.org) |
| A team of AI assistants that grade each other's work and can pause or redirect a study investigated quantum spins in diamond. When a drifting signal made the test unreliable, they called it inconclusive, fixed the measurement plan, and confirmed their theory. |
|
3 points by Betelbuddy 17 days ago | hide | past | pdf | 1 comment
|
| More |