ML Newsnew | past | best | rss | submit about
The Measure of Intelligence: towards more human-like artificial systems (arxiv.org)
80 points by yamrzou on Nov 15, 2019 | hide | past | pdf | 12 comments on HN

In plain words: Judging a system by how well it plays one game can be faked with endless practice, so intelligence is how efficiently it picks up new skills. A new puzzle test uses only the basic knowledge humans are born with to compare people and AI fairly.

Abstract · On the Measure of Intelligence

To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.

François Chollet
arXiv:1911.01547 · cs.AI · submitted Nov 5, 2019 · updated Nov 25, 2019
abstract · pdf · html

add comment on HN
Also discussed: Sep 2026 (4 points, 0 comments) · Aug 2026 (1 point, 0 comments) · Dec 2024 (2 points, 0 comments) · Jun 2024 (16 points, 1 comment) · Apr 2024 (2 points, 1 comment) · Jan 2023 (4 points, 0 comments) · Apr 2021 (2 points, 0 comments) · Nov 2019 (2 points, 0 comments) · Nov 2019 (2 points, 0 comments) · Nov 2019 (1 point, 0 comments) · Nov 2019 (13 points, 0 comments)
emilwallner on Nov 15, 2019 [–]
François Chollet’s core point: We can't measure an AI system's adaptability and flexibility by measuring a specific skill.

With unlimited data, models memorize decisions. To advance AGI we need to quantify and measure skill-acquisition efficiency.

I made a summary about it here: https://twitter.com/EmilWallner/status/1193968380967043073 Announcement tweet: https://twitter.com/fchollet/status/1192121587467784192

It's a must-read for anyone who's interested in AGI.

reply
joe_the_user on Nov 15, 2019 | parent [–]
Interesting: "To avoid local-generalization systems that artificially "buy" performance on a specific task, Chollet restricts priors to 'Core Knowledge' found in developmental science theory: such as elementary physics, arithmetic, geometry and a basic understanding of intentions."

I can see the plausibility of this but it raises an objection that the hardest tasks for a computer are those seemingly simple for a human being while tasks involving analysis and abstraction are often easiest for computers, Moravec's "law"[1]. I think one could also put it as solving neat problems is simple for machines, solving the messy intersection of several problems and exigencies is what's hard (and what's required for generality, since most things turn out to be messy).

[1] https://en.wikipedia.org/wiki/Moravec%27s_paradox

reply
xamuel on Nov 15, 2019 | parent [–]
That core point is like a variation on a lemma in a paper I recently published (Lemma 2 in [1]): There can be no single deterministic interactive reward-giving environment which can, by its lonesome, serve as a good proxy for general intelligence of deterministic agents.

The proof is fairly simple. Suppose E were such an environment. I claim that every intelligent agent is exactly as intelligent (according to intelligence as measured by performance in E) as some "blind" agent, where by "blind" I mean an agent which totally ignores its surroundings.

Let A be any deterministic agent. If we were to place A in the environment E, then A would take certain actions, call them a1, a2, a3, ..., based on A's interaction with E.

Now define a new agent B as follows. B totally ignores everything and instead just blindly takes actions a1, a2, a3, ...

By construction, B acts exactly the same as A within E, therefore, as measured by E-performance, A and B are equally intelligent. So, for any particular environment, every agent is just as intelligent as some "blind" agent. But that's clearly a very bad property for an alleged intelligence measure to possess!

[1] https://philpapers.org/archive/ALEIVU.pdf

reply
elcomet on Nov 15, 2019 | parent [–]
I'm not sure I follow.

What if you change the initial conditions of the environment?

Then A would act differently, but B would still act the same because it is blind.

So now they are not taking the same actions. It seems impossible to have a blind agent that follows a non-blind in all situations.

reply
xamuel on Nov 15, 2019 | parent [–]
The argument is for deterministic environments--no initial condition dependence.

If initial conditions are allowed, then you can consider the following environment: initial condition is an arbitrary source-code; the environment proceeds to implement said source-code. Clearly this "environment" is too all-encompasing to be considered as a single environment.

reply
aristus on Nov 15, 2019 | parent [–]
Isn't that basically Searle's Chinese Room argument?
reply
xamuel on Nov 15, 2019 | parent [–]
Not really, unless I'm missing something. They have thematic similarities, I suppose.
reply
vonnik on Nov 15, 2019 | parent [–]
There's a lot of interesting research focused on this problem space, training algorithms toward a more flexible problem-solving ability. Here are a couple:

https://arxiv.org/pdf/1902.09725.pdf https://openreview.net/pdf?id=Skc-Fo4Yg

I do think the DeepMind team is pretty aware of how well their algorithms adapt to new problems. That has been a theme since the Atari paper.

reply
XuMiao on Nov 16, 2019 | parent [–]
I am not sure that I get the definition of skill right. Does classifying hot dog or not count as a unique skill or not? Or does object detection count as one skill?

Intelligence is hard to measure. It's personal and contextual. Even the IQ test for human is incomplete and inaccurate.

I would be really impressed if AI can provide a better theory than the multi-verse to explain the quantum mechanics. I consider that the moment of true AI.

reply
sanxiyn on Nov 15, 2019 [–]
The code is at https://github.com/fchollet/ARC.

There are 1000 tasks, all unique(!), and apparently all handwritten(!) one by one.

reply
js8 on Nov 15, 2019 [–]
I am surprised he doesn't mention Bongard problems: http://www.foundalis.com/res/diss_research.html
reply
yamrzou on Nov 15, 2019 [–]
It’s worth noting that the author is François Chollet, the creator of Keras.

This seems to be what he has been working on for the past two years.

reply
About | RSS | RSS (all) | HN arXiv