about
Is Complexity an Illusion? (2024) (arxiv.org)
4 points by the-mitr 28 days ago | hide | past | pdf | 1 comment on HN

In plain words: Simple-looking models don't generalize better because they're simple; what helps is that they allow many possible causes, and goal-driven processes like natural selection just make those loose rules look simple. Picking loose rules over simple shapes improved generalization by 110-500%.

Abstract · Is Complexity an Illusion?

Simplicity is held by many to be the key to general intelligence. Simpler models tend to "generalise", identifying the cause or generator of data with greater sample efficiency. The implications of the correlation between simplicity and generalisation extend far beyond computer science, addressing questions of physics and even biology. Yet simplicity is a property of form, while generalisation is of function. In interactive settings, any correlation between the two depends on interpretation. In theory there could be no correlation and yet in practice, there is. Previous theoretical work showed generalisation to be a consequence of "weak" constraints implied by function, not form. Experiments demonstrated choosing weak constraints over simple forms yielded a 110-500% improvement in generalisation rate. Here we show that all constraints can take equally simple forms, regardless of weakness. However if forms are spatially extended, then function is represented using a finite subset of forms. If function is represented using a finite subset of forms, then we can force a correlation between simplicity and generalisation by making weak constraints take simple forms. If function is determined by a goal directed process that favours versatility (e.g. natural selection), then efficiency demands weak constraints take simple forms. Complexity has no causal influence on generalisation, but appears to due to confounding.

Michael Timothy Bennett
arXiv:2404.07227 · cs.AI · submitted Mar 31, 2024 · updated May 30, 2024
abstract · pdf · html · Accepted for publication in the Proceedings of the 17th Conference on Artificial General Intelligence, 2024. Definitions shared with arXiv:2302.00843

add comment on HN
Also discussed: Jun 2024 (2 points, 1 comment)

I couldn't tell from the abstract, do they define what they mean by complexity?