about
Chinchilla Scaling: A Replication Attempt (arxiv.org)
2 points by chewxy on Apr 17, 2024 | hide | past | pdf | discuss on HN

In plain words: They retried one of three ways to find the best split between model size and training data, fitting a curve to points read off graphs. The original numbers contradicted the paper's other estimates and had error bars needing over 600,000 experiments; the fix agrees.

Abstract · Chinchilla Scaling: A replication attempt

Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. In contrast, our rederivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al.

Tamay Besiroglu, Ege Erdil, Matthew Barnett, Josh You
arXiv:2404.10102 · cs.AI, cs.CL · submitted Apr 15, 2024 · updated May 15, 2024
abstract · pdf · html

add comment on HN
Also discussed: Apr 2024 (124 points, 68 comments) · Apr 2024 (6 points, 1 comment)