about
Understanding Scaling Laws for Recommendation Models (arxiv.org)
3 points by BachToTheFuture on Aug 28, 2022 | hide | past | pdf | 2 comments on HN

In plain words: They measured how a recommendation model that predicts clicks improves as you add parameters, data, or compute, and found quality climbs steadily but with shrinking gains. For the architecture tested, growing the model has stopped paying off, so adding more data is the way forward.

Abstract

Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and developing efficient system infrastructures to support large-scale models. In this paper, we study empirical scaling laws for DLRM style recommendation models, in particular Click-Through Rate (CTR). We observe that model quality scales with power law plus constant in model size, data size and amount of compute used for training. We characterize scaling efficiency along three different resource dimensions, namely data, parameters and compute by comparing the different scaling schemes along these axes. We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward. The key research questions addressed by this study include: Does a recommendation model scale sustainably as predicted by the scaling laws? Or are we far off from the scaling law predictions? What are the limits of scaling? What are the implications of the scaling laws on long-term hardware/system development?

Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, Adnan Aziz
arXiv:2208.08489 · cs.IR, cs.LG · submitted Aug 17, 2022
abstract · pdf · html

add comment on HN

Understanding scaling Change. How about time to change [email protected] to [email protected]?

I just did my high school graduation project on Recommendation models, this arxiv paper is a bit too specialized, anyone can give some feedback?

I found the original paper through this roundup:

https://dblalock.substack.com/p/2022-8-21-arxiv-roundup-recs...