With only a few exceptions, ML is incredibly simple (there is no AI). The math is simple, the mechanics of evaluating it is simple, the reason it works is simple, and it only really works well if you have absurd amounts of data and CPU time.

Physics is... determining the mathematics you need to know on the fly while discovering and explaining many phenomena. You can spend decades focusing on matrix multiplications and other fairly straightforward trivia to analyze your particle trajectories, and then suddenly, you need to know group theory or some completely different field of math just to understand the basic modelling.

Personally I think the most impressive thing in physics and stats so far is our ability to predict the trajectories of solar system objects far into the future. After some very serious numerical analysis over the past 50 years, we've reached the point where there aren't many improvements we can make, and most of them come from identifying new objects, their position, and mass (data/parameters), and the real argument is about whether the underlying behavior is truly unpredictable even if you have perfect information.

Of course, the last best work in this area was done by Sussman who has been an ML researcher for some time: (https://www.researchgate.net/publication/6039194_Chaotic_Evo...)

As you can see, physicists pretty much invented all the math to do ML whilst solving other problems along the way: https://en.wikipedia.org/wiki/Stability_of_the_Solar_System

in fact many of my friends who were physics people, when I show them the code of a large scale batch training system they wonder why they did physics instead of CS because the math is so unbelivably simple compared ot the tensor path integrals they had to learn in Senior Physics.

the author is talking about constraint optimization in physics vs ml/ai

constraint optimization is easier with a well defined prior vs the guess work in ml algos.

dl models in physics can scale and emergent phenomena will depend on scaling parameters -- this is coarse graining in physics.

> my graduate work was using constraint optimizers in molecular dynamics

me too for qm/mm sims -- did some rg/complex systems work too ;)

http://proceedings.mlr.press/v76/ensign17a/ensign17a.pdf

"The key idea behind our results is that if we can find the symmetries of a problem then we can solve it"

I appreciate that symmetries are great but that doesn't solve a problem.

"As a first step in that direction, we discuss an upcoming book on the principles of deep learning theory that attempts to realize this approach.

Comments: written for a special issue of Machine Learning: Science and Technology as an invited perspective piece"

So take it for what it's worth.

[0] https://deeplearningtheory.com/PDLT.pdf

Any simplicity observed in physics is born of long familiarity, or is the result of underlying complexities being masked or approximated away.

We also don't know for sure the "constants" are constant throughout the universe (spatially and temporally) This is assumed for now, and seems almost certainly true.

I think it's unsafe to assume the above are Absolutely True and that that's why physics is simple.

This is a much more nuanced matter. Non-locality as in global wave function collapse or Bohmian mechanics has not the same consequences as classical non-locality, there is no causal influence from things you do to an entangled particle at the other end of the universe. Also to entangle particles they have to first interact locally before they can be separated.

We also don't know for sure the "constants" are constant throughout the universe (spatially and temporally) This is assumed for now, and seems almost certainly true.

This does not really change the argument, even if, for example, the fine-structure constant is not constant after all, then there will most likely be a hand full of other constants that describe how it varies over space and time. This is very different from every electron having a unique electric charge that is not governed by anything and the only way to figure it out is to measure it for each electron.

It would also probably make a difference how values are distributed, are the charges of the electrons nicely distributed and vary by a factor of two or ten? Some statistical theory could probably deal with that. But what if there is no nice distribution, if values vary by hundreds of orders of magnitude, if the expectation value or the variance of the charge is infinite? I am certainly unqualified to make any definitive statements but I can imagine physics to be weird to the point that it becomes mathematically intractable or at least only produces useless answers because of the amount of uncertainty that enters the equations.

In any case, we know that the universe is simple to a very good approximation, even if fundamentally everything depends on everything and the electron charges are all random, those effects are small and we can have a good approximation with simple theories.

"Please note that the notion of simplicity to which we are appealing is not at all meant to suggest that physics is trivial. Instead, we mean it as a compliment: we have the utmost respect for the work of physicists. This essay is an apologia for physicists doing machine learning qua physicists; it is meant to interrogate what it is about the approach or perspective of physicists that allows them to reach so far in explaining fundamental phenomena and then consider whether that same approach could be applied more broadly to understanding intelligence"

edit: added quotes

General relativity, quantum chromodynamics, etc. They are all incredibly complicated.

And e=mc^2 is not a simple formula. Okay, it's simple. But deriving it and understanding why it's like that was what, junior year in college?

It was a while back but I think we derived it in school aged 16-17 in first year A level Physics (UK.) Is that roughly "junior year in college" elsewhere?

Edit: Oxford University, since you are from the UK and that's reputable within the country, lists General Relativity (and hence E=MC^2's derivation) as a 3rd year course in the Physics program. So that makes it squarely in the junior year.

I consider that a big chunk of the world.

which might not be m*a but usually is.

physics is also mostly deterministic (attach probability distributions for stochastic/quantum stuff) and there are well defined rules (energy, symmetry, noether, etc).

at the end of the day ai has some space for models and so does physics. because physics has well defined rules it's easier to apply constraints to that space vs ai/ml where it's informed guesswork.

of course there will be a correspondence between parameters in a model and emergent physical phenomena ... and i'm sure really nice scaling laws, etc will come out , this is just coarse graining.

onsager and the likes were onto this stuff way before deep learning was a thing. i think this connection is uninteresting because optimization in it's heart is physics. dl is just one aspect.

- a physicist (escaped to the greener pastures of swe, shame really, i miss it but not the wl balance)

On the content: it's pretty weird that our best models don't use much of the world's underlying structure at all. State-of-the-art vision models like vision transformers and MLP-mixer do just fine when you shuffle the pixels. You could argue that modern image datasets are so big that any relevant structure could be learned by attention, but it still feels like we're doing _something_ wrong when pixel order doesn't matter at all ¯\_(ツ)_/¯

[1] https://danintheory.com/

They won't generalize after training to transformations of the data that they haven't been trained on, even simple ones such as rotations, whereas humans will.

So I would argue that vision model do use the "underlying structure", and even that one of their problems is that they make use of some of the "underlying structures" that are not actually important, such as image luminosity, rotations etc. I think people usually augments the data with these transformations beforehand during preprocessing to enforce invariance.

AI is figuring out how to replicate something fuzzily understood, from a difficult to test domain, using techniques that are largely in a different domain altogether, because even for the parts of the inspirational domain we kinda-sorta understand, we don’t have the tools to directly reproduce them easily so at best we simulate them in alternative media.

(“Physics is simple” overstates the case, still, but “AI is hard” understates the case, so, relatively speaking, its something of a wash.)

        This is why (understanding how) AI (works) is hard.

        This is why physics is (able to offer) simple (explanations for seemingly complicated natural phenomena) 

1. "top-down analysis" to deep learning is ineffective due to the nature AI.

2. It should be possible to use the "sparsity" principle from physics as "a guiding principle of deep learning theory"

Where the "sparsity" principle dicit:

        The number of parameters defining a theory should be far less than the number of experiments that such a theory describes

Personally, I'm skeptical. I prefer to understand whole theater of ANN as an attempt to optimize neural-net for practicality. Even deep learning is a technique for structuring the network for efficient training.

I thinks so because even the most dumbest neural network - layers with high connectivity - can perform certain tasks, if we train it really really hard with fingers crossed. To avoid uncertainty and improve performance, it's necessary to structure the network. This cuts unnecessary computation, and prevents over-fitting and over-labeling.

That is, we have to inject bias to network, and this makes any bottom-up analysis biased and less generalizable.

For example recognising pictures of cats, the fact you can rotate a cat picture and its still a cat isn't necessarily baked in. If you visualise layers there seems to be a lot of redundant information, (if you train on rotated cats you get layers that look like rotations of each other). If there is a mechanism to store this kind of information more compactly, I am not aware of it.

Perhaps there is the idea that some kind of structure that encodes the redundant information, should be emergent or natural, it should evolve out of the models without having to make arbitrary choices. On the other hand maybe we should turn to physics and add these physical kind of constraints, there is a way to renormalise or make use of symmetries to improve our current models. Perhaps a neural network is more like an algebraic variety (collection of polynomials) with a group action.

We've known the principles of Newtonian physics since, well, Newton.

We'll need an Isaac-Newton-level of insight into intelligence before we can make it so simple.

Nevermind that we can't even seem to agree on the definition of intelligence.

As to the definition, I have an objection to calling the current big-data stats we do nowadays "machine learning" or "artificial intelligence". There is no intelligence there.

Rather, to allude to the article, I consider it more "machine muscle memory" or "artificial intuition". The algorithm can tell you, "I have a really good hunch about this based on the zillions of examples I've seen" but it can't derive underlying truths to reason about why.

Perhaps we are recapitulating phylogeny when it comes to artificial neural systems? We have something like an autonomous nervous system, and a brainstem, but we need so much more to get to intelligence.

Physics simple?

Physical intuition?

The single most successful program of physics is quantum mechanics, and it is neither intuitive nor simple. Relativity, while conceptually simple, isn't so simple either and is far from intuitive (consider that momentum is conserved during a relativistic near-miss collision only if one considers the entire collision over a sufficiently length period, since at any moment the force vector between the bodies does NOT align with separation vector, since the force acting at one moment was exchanged, at c, when the bodies were in different positions).

There are a lot of simple concepts in physics, many of them basically teaching aids to get people started. When one gets deep into the field, simple and intuitive go by the wayside.

Let's get physicists to look at AI so we might make some progress, btw here's a new book that tells us how

I'm not saying that's what the article is actually about but that's what I read from it and it's crass enough that I didn't read further.

We forget at our peril the Michelson-Morley Experiment and the Ultraviolet Catastrophe, and if we forget these, we may assume now too that we have it all figured out.

Of course, active researchers in the subject, both theoretical and experimental, do not forget.

Based on the parts which I've read so far, a more accurate title would be 'Why some currently hot parts of AI not well understood, and some parts of Physics well understood?'

I think the original title is an ok approximation of this.

However, the true difficulty in physics is arriving at that model in the first place. Decades of work offered up against experiment, the associated conceptual leaps in understanding required to get to e.g. a quantum field theory which succesfully predicts things are nothing short of a monumental achievement. To say that physics is simple is ludicrous.

Author is talking about how our physical theories, such as QFT, currently have more predictive power than any theories we currently have about machine learning/deep learning.

(Author has a PhD in theoretical physics).

In practice, the computations required to use the QFT model are just too complex for modern computers when it comes to single atoms with more than a few protons, not to mention larger molecules. Instead, we must use simplified models like the Bohr model to make predictions about molecular bonds.

This actually seems to be very similar to AI where we understand not everything, but a lot about basic neurons, yet the emergent phenomena of intelligence is very difficult to predict due to the explosion of computational complexity.

You will notice the article conflates the two things: physics and the known laws of physics (e.g. first para in section 1.2). Simplicity of the latter does not imply simplicity of the former, but the article assumes that it does in order to tackle/state the question as posed: "Why is AI hard and physics simple?".

Starting with early calculus, to differential geometry, Hilbert spaces and whatnot, the brain doesn't fit models to data, it is making up categories and concepts (new classes of metamodels if you wish) as it goes along (as improved experimental devices augment our sensory inputs). To cope and explain these totally indirect information flows the brain conjures up symmetries in invisible abstract spaces, invariants and visual imagery from alien internal worlds, pursues "thought experiments" to restore sanity... Machine learning style fitting of "model" to data is just one of the final steps. Important but hardly defining the process.

The "AI" folks oversold their goods by a breathtaking factor and are now exposed to the entire planet. No physicist will ever be able to bail you out :-)

Non-practitioner in either field ...

If you don't focus on the more or less "defined" ML subset, and focus on the AI in the question, whatever "AI" has meant over the years, then ...

The reason why AI is hard and physics simple may be that physics already "is," while we may not yet have defined what AI should be and what to do with it .

Physics has guardrails of reality, however subtle and hidden they may be. What is described by physics is what it is, and will be there as it is, whether we correctly describe it or not, and whether we observe it or not. (Depending on your view of an observed universe's dependant relationship with its observer.)

AI is ... what? In any particular decade? We aren't describing anything that already is, and that comes with constraints. We're making it up.

Specification is hard.

The universe as described by physics, on the other hand, is well beyond specification and requirements gathering. We're just writing the missing manual.

AI is a black box you whack with a stick until you like the results.

https://vixra.org/

AI is not yet common knowledge. It isn't as well understood.

But you just wait.