ML Newsnew | past | best | rss | submit about
The Mathematics of Artificial Intelligence (2022) (arxiv.org)
181 points by ofou on Apr 11, 2022 | hide | past | pdf | 59 comments on HN

In plain words: It reviews the math used to explain deep neural networks, the layered pattern-matching systems behind most of today's AI. It lays out the main theoretical directions, a few proven results, and the questions still open.

Abstract · The Mathematics of Artificial Intelligence

We currently witness the spectacular success of artificial intelligence in both science and public life. However, the development of a rigorous mathematical foundation is still at an early stage. In this survey article, which is based on an invited lecture at the International Congress of Mathematicians 2022, we will in particular focus on the current "workhorse" of artificial intelligence, namely deep neural networks. We will present the main theoretical directions along with several exemplary results and discuss key open problems.

Gitta Kutyniok
arXiv:2203.08890 · cs.LG, math.HO, stat.ML · submitted Mar 16, 2022
abstract · pdf · html · 16 pages, 7 figures

add comment on HN
Also discussed: Mar 2022 (4 points, 0 comments)
mg on Apr 11, 2022 [–]
After a cursory glance, my feeling is that "The Mathematics of Neuronal Networks" would be a better title?

In recent years, the term "Artificial Intelligence" is often used instead of "Neuronal Networks".

I wouldn't be surprised if this will change again. If there is evidence that it will not and Neuronal Networks are for some reason the optimal medium for intelligence, I would love to read about it.

reply
plafl on Apr 11, 2022 | parent [–]
I like to explain that AI > ML > NN.

Now, NNs are the ones getting results at computer vision and natural language, and more. I think most people would say that other ML approaches are computational statistics. The goalpost for AI keeps moving.

If you are truly interested in the math of AI I think PAC Bayes learning is more appropriate and your book is Understanding Machine Learning [1] (not an easy read). A more gentle intro would be Learning From Data [2]. If someone recommends a book/paper it would be awesome, I'm always on the look.

[1] https://www.cs.huji.ac.il/w~shais/UnderstandingMachineLearni... [2] https://work.caltech.edu/telecourse

reply
kvathupo on Apr 11, 2022 | parent [–]
Although it tends towards Deep Learning as opposed to AI, I highly recommend Bishop's Pattern Recognition and Machine Learning. It not only provides a solid Bayesian perspective, but also comments on the subtleties of applying theory. For the latter, its discussion of overparameterization in (I think?) the first chapter comes to mind.

[1] - https://www.microsoft.com/en-us/research/people/cmbishop/prm...

reply
akomtu on Apr 11, 2022 | parent [–]
NN > ML. The proof is that nematode or fly, I dont remember, with its simple NN fully mapped, and still remaining a mystery how it works. ML, which is just a matrix multiplication at its core, is a laughably simplistic model of NN.
reply
soVeryTired on Apr 11, 2022 | parent [–]
GP means artificial neural network, not an actual nervous system.

Machine learning contains ANNs as a sub-discipline. Other non-ANN topics in ML include ensembled trees, Gaussian processes, and sampling theory.

reply
kvathupo on Apr 11, 2022 | parent [–]
I think one can't compare deep learning to machine learning since they have different purposes. While there have been great strides in the interpretability of NNs, the analytic models of classical machine learning are favored from a computational and interpretability perspective. Conversely, if the relationships in our data are constantly evolving (e.g. the change in the stochastic process followed by a time-series from one interval to another), then NNs are more appropriate.
reply
jjtheblunt on Apr 11, 2022 | parent [–]
linear-algebraic matrix multiplication complicated by non-linear threshold discontinuities evaluators, no?
reply
MrBusch on Apr 11, 2022 | parent [–]
Seconding this. "Learning From Data" is one of the best intros to machine learning theory I've seen through the years!
reply
martopix on Apr 11, 2022 | parent [–]
Because AI basically "stole" the term neural networks from neuroscience, neuroscientists now often speak of "neuronal networks" while referring to biological ones, to avoid confusion. So I would recommend to not use "neuronal networks" in this context.
reply
paskozdilar on Apr 11, 2022 | parent [–]
This is a very good point. To anyone interested in what Artificial Intelligence looked like before neural networks, the "Artificial Intelligence - A Modern Approach" by Stuart Russell and Peter Norvig is a good book on the subject.
reply
kriro on Apr 11, 2022 | parent [–]
Agreed. I was teaching from it (AI 101, search/agents, no ML) and it's possibly the best textbook I have encountered. It's structured very well.

As an aside, I will forever remember the rough geography of Romania. I still plan to do an A* route visit of the country some day :D

reply
mdp2021 on Apr 11, 2022 | parent [–]
> Neuronal Networks are for some reason the optimal medium for intelligence

No, that's not the same 'intelligence' ("General Intelligence") as the "I" side in Artificial Intelligence.

The term 'intelligence' applied to Artificial Neural Networks makes sense, as such: to reach a procedural solution it takes an engineer; the engineer is said to have reached the solution because "intelligent"; ANNs are (semi-)automated builders of function approximators; ANNs are said "intelligent" because they "reach solutions" (like the engineer would have done - "approximator":"engineer"="Artificial":"Natural" Intelligence). And of course they are not "intelligent", while they are in some sense. It's just an expression, it's rhetoric (it requires considerate interpretation).

That some sort of "intelligence" is achievable in other ways, or that ANNs spawn as an idea from anatomical considerations of naturally intelligent entities, or that ANNs could help in modelling general intelligence (etc.), is tangential.

reply
akomtu on Apr 11, 2022 | parent [–]
ANNs dont have the key component of intelligence - ability to learn. If our brains worked like ANNs, they'd be wired once at the brains factory, they'd produce the same output given the same input, and learning anything new would need a complete rewiring at the factory, possibly erasing other learnings (often unpredictably).

Even if ANNs could update themselves on the fly, in a reasonably incremental manner, they'd just imitate insect-level reactions. Intelligence would need those ANNs to have a virtual reality, run thought experiments there and learn from that.

reply
sdenton4 on Apr 11, 2022 | parent [–]
You're asking for online learning. This is certainly possible with modern techniques - define an unsupervised loss and update the weights with gradients as new data is encountered. Federated learning also works along these lines.

'Raw' online learning is unpopular because you have no garauntee that the network won't do something funky in the field.

That said, I think there are production systems in the world which learn on a day by day basis. Eg, take all of the logs from the last day and use them to update the production model for tomorrow. Then there's enough data that you don't risk a bad step. Think of it as learning while dreaming...

reply
mdp2021 on Apr 11, 2022 | parent [–]
Well this does not mean that some mechanisms similar to those we are studying cannot be part of the original or that they cannot be part of a General Intelligence system. Even if that were simply "adjusting a structure to function and information retention". The very "compositional" nature of the perceived that makes Convolutional ANNs work was noted after biomimicry.

This, of course, as an aside.

reply
DataDaoDe on Apr 11, 2022 | parent [–]
Came here to say the same thing. This is not a paper on mathematical artificial intelligence, at last as I understand it, not in the way Goertzel or Hutter write about it. I was actually thinking this would be something like Hutter's universal artificial intelligence.
reply
elcomet on Apr 11, 2022 | parent [–]
We usually say "neural networks" and not "neuronal networks"
reply
mdp2021 on Apr 11, 2022 | parent [–]
I think in Greek 'neura' and 'neuron' are equivalent. The perception that '-on' indicate the single or ideal entity ("ion, electron, muon...") comes from Whewell and Faraday ('-on' ex "ienai": the "goer" particles) and fully postdates the use of 'neuron'.

Edit: also see the comment from member martopix, nearby ( https://news.ycombinator.com/item?id=30988150 )

reply
tomrod on Apr 11, 2022 | parent [–]
You have the correct etymology, but the convention is for "Neural Network" in ML and "Neuronal Network" in Neuroscience for we laymen.
reply
mdp2021 on Apr 11, 2022 | parent [–]
As noted in the reference to the post from martopix :)
reply
melling on Apr 11, 2022 [–]
I’m on Chapter 4 of the Mathematics for Machine Learning. Enjoying it so far:

https://mml-book.github.io/

After learning the math, which machine learning books come next?

reply
pa7x1 on Apr 11, 2022 | parent [–]
If you are looking to have a mathematical understanding of the field then I really enjoyed "Understanding Machine Learning: From Theory to Algorithms" by Shalev-Shwartz and Ben-David.
reply
wrp on Apr 11, 2022 [–]
When you compare it to Edward Bender's old book[1], the "math of AI" seems to have become a very narrow subject.

[1] https://www.amazon.com/Mathematical-Methods-Artificial-Intel...

reply
mdp2021 on Apr 11, 2022 [–]
Does anybody know of textbooks, or articles, material etc. that focus on the /consequences/ of the mathematics of ANNs? For example: why in this problem using more layers is more or less efficient than using bigger layers, or branching evolutions of the outcomes as opposed to intensive computation of a single flow etc.

A reasoned summary of the tricks, in a way ("this works because of that").

reply
nuclearnice1 on Apr 11, 2022 | parent [–]
Some of the questions of depth vs breadth are explained intuitively in this Bengio Lecun paper from 2007.

http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf

reply
alok-g on Apr 11, 2022 | parent [–]
Check this out:

https://ai.facebook.com/blog/advancing-ai-theory-with-a-firs...

HN discussion:

https://news.ycombinator.com/item?id=27559017

My own review comment:

https://news.ycombinator.com/item?id=27564506

Edit: Corrected the third link. Thanks mdp2021 for notifying me of the copy-paste error.

reply
mdp2021 on Apr 11, 2022 | parent [–]
Thank you alok-g,

I would just correct the links:

-- article: https://arxiv.org/abs/2106.10165

-- book: https://arxiv.org/pdf/2106.10165

-- your summary: https://news.ycombinator.com/item?id=27564506 (the link to the post is that "under" the date. You posted two identical links here)

reply
informal007 on Apr 11, 2022 | parent [–]
Agree
reply
woopwoop on Apr 11, 2022 [–]
Thanks, this is great.

Poorly thought out morning spitball coming. One of the reasons approximation theorems are so unsatisfying is that they are always of the form "for function class X there exists an architecture A of complexity O(N) such that blah". And then this is compared favorably with some other function class whose dimension is O(N). But there's something tricky about this: you leave the architecture unspecified. You are comparing a single space of functions with an enormous number of spaces, one for each architecture with the specified complexity, and then saying "well if I pick the right architecture I win". Doesn't seem like a fair comparison.

reply
mistrial9 on Apr 11, 2022 | parent [–]
more spitball -- the term "AI" changes the context from machine-side to application-side. "AI" implies to the listener that the machine is doing something that only thinking humans could do.. emphasis on the humans, while sets of math that seek minimized error, emphasize the behavior of the functions.
reply
dpflan on Apr 11, 2022 [–]
Along the same lines, here is a course on the "Mathematical Foundations of Machine Learning" [1] (from GT, but other institutions also having offerings) and is a section in the paper in this post (so deeper dive possible):

[1.] https://jrom.ece.gatech.edu/mfml-f20-notes/

reply
zozbot234 on Apr 11, 2022 [–]
Shouldn't "the mathematics of artificial intelligence" be subsumed by computational statistics? It may be that only very general and weak statements are possible about "deep" NN's, but then that's merely a fact about the territory we're dealing with. Why should this be regarded as a "new" field?
reply
pietroppeter on Apr 11, 2022 [–]
Link to vanity version (a web version of the pdf): https://www.arxiv-vanity.com/papers/2203.08890/
reply
mistrial9 on Apr 11, 2022 | parent [–]
who runs this site? where are the logs stored?
reply
mellosouls on Apr 11, 2022 [–]
Let L(uy,y) = fy denote a parametric partial differential equation with y being a parameter from a high-dimensional parameter space Y ⊆ Rp and uy the associated solution in a Hilbert space H. After a high-fidelity discretization...

These AI folks are clearly very clever, but they don't actually believe that's got anything to do with how human thinking works, right?

reply
akomtu on Apr 11, 2022 | parent [–]
Whenever I see "Hilbert space" remarks in a paper that doesn't concern abstract math, I roll my eyes and imagine a chef who starts his youtube video with "Today we are going to cook a salad. For better precision, we are going to use knifes of triangular shape, three dimensional plates of circular shape, and the 100% dihydrogen of oxyde solution." All this only to obfuscate the fact that the entire recipe is just chopping a few carrots.
reply
martopix on Apr 11, 2022 | parent [–]
Try doing quantum mechanics, even at beginners level, without a proper knowledge of Hilbert spaces.
reply
nancarrow on Apr 11, 2022 | parent [–]
the subclass of "Reproducing kernel Hilbert spaces" form the basis for a large class of ML algorithms (kernel methods)
reply
martopix on Apr 11, 2022 | parent [–]
It's literally called "the mathematics of ARTIFICIAL intelligence"
reply
mellosouls on Apr 11, 2022 | parent [–]
The important word is intelligence.
reply
mdp2021 on Apr 11, 2022 | parent [–]
Research on natural intelligence or general intelligence presents itself as such.
reply
mellosouls on Apr 11, 2022 | parent [–]
AI leaders occasionally seem to pontificate on AGI, yet it's difficult for a layperson like me to see how their expertise (as seen in publications like this) translates.
reply
WithinReason on Apr 11, 2022 [–]
Is it correct to use "we" if there is only one author?
reply
rcap on Apr 11, 2022 | parent [–]
It's standard practice for mathematics papers. Some people like to view it as "we, the author and the reader".
reply
mdp2021 on Apr 11, 2022 | parent [–]
A form of "plural majestatis" which paradoxically implies that the individual author has only relative importance

(«paradoxically» as the majestas itself is founded on representing something higher, not of some high status of the individual which happens to be endowed - it may sound like boasting but on the contrary it would be a downplay of the individual).

In other contexts, that "we" can have even more foundations: "I could never have done this without the work of others", "Not just me but all those who think alike" etc.

reply
woopwoop on Apr 11, 2022 | parent [–]
Using the "royal we" is standard in mathematics. If you ever see someone do this in ordinary writing you can bump up your odds that they're a mathematician. Another giveaway is using "modulo" to mean "except for".
reply
hatmatrix on Apr 11, 2022 | parent [–]
It's the "royal 'we'"
reply
Iwan-Zotow on Apr 12, 2022 | parent [–]
me and cockroaches in my head
reply
pietroppeter on Apr 11, 2022 [–]
Pity that it uses Artificial Intelligence instead of the more correct and precise term Machine Learning, especially since Mathemticians are supposed to be more precise.
reply
mdp2021 on Apr 11, 2022 | parent [–]
It is mathematically correct: what concerns ML concerns AI.

The interest of the author is explicitly ("precisely") on AI, where, statedly, «the current “workhorse” of artificial intelligence [are] namely deep neural networks».

reply
dr_dshiv on Apr 11, 2022 [–]
Has anyone followed Smolensky’s Harmonium approach? I think it is so cool that “goodness of fit” was based on “harmony”…
reply
etiam on Apr 11, 2022 | parent [–]
Have a search under the alternative name Restricted Boltzmann Machine if you haven't. Several interesting developments since the 80s.
reply
dr_dshiv on Apr 11, 2022 | parent [–]
Oh yeah well aware that it was the first restricted Boltzmann Machine. It minimizes energy, but that’s just harmony with a sign change. When Smolensky Hinton and Rummelhart collaborated, they decided to call it “goodness of fit.” So neural networks still optimize for harmony by another name. I think that’s so cool. The original Smolensky paper is worth reading. Good ideas.
reply
ThereIsNoWorry on Apr 11, 2022 [–]
I finished my studies in CS less than a year ago. Nothing about this paper is new, but it gives a quick overview.
reply
Sirenos on Apr 11, 2022 | parent [–]
No need to be snarky. It could be new for someone just getting into the field.
reply
mdp2021 on Apr 11, 2022 | parent [–]
The poster probably only meant to convey that the article is compilative (a summary, as opposed to new research) for those who are up to date.
reply
ImprobableTruth on Apr 11, 2022 | parent [–]
Is "Nothing about this paper is new" an appropriate response when the abstract explicitly mentions that it's a survey paper? Maybe I'm just sensitive, but to me that sounds dismissive rather than purely informative.
reply
mdp2021 on Apr 11, 2022 | parent [–]
Well, the poster may have just rephrased the formula («survey article»), or may have noted it for the benefit of the skimmer, or just used it for the conclusion out of their judgement "for an overview it does its job". «sensitive ... sounds»: the author is apparently a native German speaker and not only different minds and cultures weigh expressions differently: there could even be a linguistic bias. We are mandated and obliged to «Assume good faith».
reply
diffeomorphism on Apr 11, 2022 | parent [–]
> Nothing about this paper is new, but it gives a quick overview.

Just as the paper itself clearly states. So your comment comes across as rather rude.

reply
About | RSS | RSS (all) | HN arXiv