about
A Simple Framework for Contrastive Learning of Visual Representations (arxiv.org)
1 point by sonabinu on Feb 16, 2020 | hide | past | pdf | discuss on HN

In plain words: It teaches a network to recognize when two altered versions of the same photo belong together, without labels, and tests which ingredients make that work. A classifier on the learned features hit 76.5% accuracy, 7% better than self-supervised methods and matching a supervised one.

Abstract

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton
arXiv:2002.05709 · cs.LG, cs.CV, stat.ML · submitted Feb 13, 2020 · updated Jul 1, 2020
abstract · pdf · html · ICML'2020. Code and pretrained models at https://github.com/google-research/simclr

add comment on HN
Also discussed: Feb 2020 (2 points, 0 comments)