The interesting data are in "Table Supp. 1" at the bottom of page 14.

I opened the paper hoping to see some examples of images that look to me like one thing on first glance, and something else on closer inspection. The best image is the one with the spider on a blurred-snake background, and that's not going to trick anyone who looks at it for more than a second.

The humans were shown each image for either 63ms or 71ms. That's 1-2 frames of a movie. So whilst the result is important, it's not as surprising as you might expect.

To be fair to the computers, that’s about the speed we train image classifiers to run at, if not faster. It seems likely that if we were willing to tolerate a second or two of latency we might be able to devise architectures that are able to see through some of the images that confuse current architectures.

The problem is not that we're not willing to tolerate latency. The problem is that the model of how neural networks are trained is completely different from how humans learn to see. When a neural net is trained, it is shown a static image, weights are tweaked until the output is correct, and then it is shown a completely different static image and the process is repeated. Neural net learning is iterative, discrete, and supervised. Human visual learning, by contrast, is continuous and largely unsupervised. We don't see snapshots, we see continuously varying images. Furthermore, we actively interact with the world by manipulating objects and shifting our gaze, and that information is also incorporated into our visual learning. Finally, humans have very advanced feature detectors built in to our brains by evolution. We don't learn to see cats, we have cat-detectors built in to our brains by our DNA, which learned to detect cats because that was a useful skill in our ancestral environment, when cats were a lot bigger and could eat us. We do learn that the thing that our cat-detectors detect is called a "cat", but we don't "learn" what a cat (or a human) looks like. That's built in to our brain wiring. (There are some things that we do learn what they look like, like cars, which obviously didn't exist in the ancestral environment. That's why all humans can tell the difference between a cat and a dog, but not everyone can identify whether something is a Honda or a Toyota.)

The point is: the process that humans go through when they learn is completely different than the process that contemporary neural nets go through. No one has yet come up with a theory that combines all of the features of human learning into an implementable algorithm. It will surely happen eventually, but there are at least a few more conceptual breakthroughs that will need to happen. Minor tweaks to back-propagation won't do it.

Do you have a citation on humans having from birth classifieds for things like cats?

I think the GP didn't mean to imply that humans have an innate cat representation from birth.

It makes more sense to interpret the comment as saying that humans don't learn an internal image representation. Humans do learn representations of bridges, aircraft, cats, etc. But those are built on top of an image processing/representation system that we are born with, analogous to raster graphics?

Edit: Maybe I'm misreading the comment. What's definitely built in at birth is things like edge and orientation detectors. A zebra detector would be a surprise.

> I think the GP didn't mean to imply that humans have an innate cat representation from birth.

Actually, that is what I meant, though I don't have a reference for cat detectors per se. But there is ample evidence for innate feature detectors of comparable complexity (e.g. human facial expressions), even if the actual target is something other than cats.

I don't know about cats, but the ability of newborns to identify human faces is well documented. The details don't really matter. Whether it's cats or something else, we humans come with some very sophisticated feature-detectors hard-wired in.

There are obviously classes of objects that we don't have hard-wired detectors for. The face is one of the few that I've heard claimed as wired from birth.

But I think "feature detectors" are exactly what the earlier comment was referring to, e.g. a Gabor-wavelet-style decomposition of the retinal image. Deep learning systems have to learn those; we're born with them.

> Gabor-wavelet-style decomposition of the retinal image

Well, that's one theory. But I think it will turn out to be a lot more complex than that. One thing that I haven't seen anyone pay much attention to is feature detectors in the time domain, which we clearly have. We notice movement as a fundamental feature. Our movement detectors can actually be triggered by static images [1]. One of the ways we distinguish dogs from cats (I believe) is by the way they move. It would be a very interesting experiment to use CGI to make a dog move like a cat and vice versa and see how those are perceived.

[1] http://www.psy.ritsumei.ac.jp/~akitaoka/ICP2016.html

It's not just the time. Computers don't have higher reasoning that can cross-check the results with expectations from a more complex world model. So by just flashing images I guess you only activate some earlier parts of the visual processing machinery. So that's quite fair to the computers since their NNs only do primitive pattern checking without reasoning too.

It reminds me of academic papers on graphics, where there is often a 50x50 pixel black and white image that looks like it was faxed back and forth 3 times. heh

Speaking of similarity between CV and human visual systems; it was kinda striking to me just how psychedelic/"trippy" the "universal adversarial perturbations" (see page 5 in https://arxiv.org/pdf/1610.08401.pdf) look.

Indeed, the look of visual hallucination "form constants" is intimately related to the machinery in the visual cortex that detects edges/contours/surfaces: https://www.math.uh.edu/~dynamics/reprints/papers/nc.pdf and such visual hallucinations can be elicited without any drugs or physiological interventions/defects -- just via diffuse flickering light: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3182860/

There's a video demo of the first paper here: https://www.youtube.com/watch?v=jhOu5yhe0rc

i was trying to build a bit of intuition about this paper by considering a trivial case where there are two classes, and the classifier is linear, in very low dimension. consider the following trivial example:

https://en.wikipedia.org/wiki/Linear_classifier#/media/File:...

in this example, the decision boundary from different classifiers is shown as H_1, H_2, H_3. The "universal" perturbation for each of the three classifiers would be a small vector normal to the each classifier's decision boundary. This paper defines "universal" perturbation with respect to the choice of input from the population of inputs, but each "universal" perturbation is optimised specifically to target a single model (aka classifier).

both H_1 and H_2 do a reasonable job of separating the two classes, but the H_1 decision boundary with the smaller margin is more vulnerable to misclassification if inputs are perturbed by a small vector normal to its decision boundary.

You can imagine translating the input space a bit to the right -- this would result in H_1 misclassifying say 2 out of the 18 data points shown, whereas H_2 (the SVM generated decision boundary with maximal margin) classifier would not experience any errors.

Interesting. Anyone in the space have some meta/context to share?

I may have this wrong but it seems the authors are very interested in a particular subset of classification errors common to machines and humans.

What is that subsets of "transferable" mistakes are trying to find? What does the existence of a particular subset of "adversarial example" (sneakily doctored images) tell us either about human or machine brains?

This is what the authors say: "A rigorous investigation of the above question creates an opportunity both for machine learning to gain knowledge from neuroscience, and for neuroscience to gain knowledge from machine learning."

Does this mean we can stop seeing adversarial examples as a deep fundamental flaw of deep neural networks? Seems like human system experiences them as well!

In the experiment, humans showed a minor drop in accuracy (<10%) after seeing the stimulus for 63ms. The presentation was so short that humans made a considerable amount of mistakes even with the non-adversarial examples. In contrast, neural networks get adversarial stimuli consistently wrong even though they are allowed to fully process them. The results also do not prove that the underlying mechanisms are the same in humans and neural networks.

Aren't optical illusions just adversarial examples for people?

No, and I wish people would stop that meme.

Humans can obviously be tricked, in a variety of ways. But adversarial images take advantage of the fact that image-recognizing neural networks do not fit their image recognition into a full fledged understanding of the world like we do. So a few pixels here and there can make a truck look like a panda and the algorithm never says, "But wait, pandas are mostly black and white and this is mostly yellow," or, "But I don't see legs anywhere, or ears."

Optical illusions mostly don't cause high level image misclassifications. To the extent that they are anything similar, they're the reverse: using our general world understanding to cause glitches in our information processing, such as cases where you think something is darker or lighter than it is, or bigger or smaller, or bent or straight. Those are your mind applying rules that are based on "how the world usually appears at a high level" to an image where those rules do not in fact apply.

I’m no machine learning expert, but it seems to me that neural networks just don’t really work like people do, as much as people would like to claim that they’ve created something that works like the human brain.

For me people work unnervingly like neural networks, except for cases where stimuli have gone through slow sequential processing by the entirety of brain computational resources, we call consciousness.

Cached responses, for example. A question in a text form, which includes a clause pointing at a flaw in usual answer, gets the same usual answer.

Effectiveness of pointing and calling technique [0], which forces full conscious attention.

And other results in psychology, which can be explained by artificial neural network level dumbness of some behaviors. (If 7 workers can build 7 cars in 7 days, then how many days would it take 5 workers to build 5 cars? Is Winnie-the-Pooh a boar or a pig?)

[0]: https://en.wikipedia.org/wiki/Pointing_and_calling

Sure. So, then, why did you posit a direct parallel between optical illusions and adversarial images?

Because that’s another thing I can now mention when people tell me neural networks are like the human brain, since this is an example they often like to bring up.

That said, one can stop before they get to optical illusions. Simple camouflage works as a similar trick. Consider stick bugs. Unless you see them move, or are directly looking at the facial features, I expect even machines will have a hard time detecting them.

We know human systems experience them as well. But we can design roads and other things so that they don't trick us too much. We have very little intuitive design understanding of what can trick robots. And in a lot of cases this is also something how you can hack an AI system if you understand what it misperceives.

As xkcd points out, you don't have to worry too much about people trying to trick self-driving systems with visual hacks.

https://xkcd.com/1958/

The XKCD comic strip makes for a useful and shallow zinger, amid casual banter, but the real world actually faces two or three new layers of complexity, which actually reassert the problem it attempts to dispose of.

1. The global reach of networked telecommunications permits a small quantity of sociopathic murderers to operate from beyond jurisdictions that can reach them, and also assists in destroying evidence of their interference.

2. Computational power enables force multiplication, such that even just one sociopathic murderer could exploit software flaws across millions of vehicles, simultaneously.

3. Some software exploits will work against self driving cars, which could never work against an ordinary person, and of course, vice versa, but not so much via remote control at a distance, when people are the operators, while we still lack electronic interfaces to our central nervous system.

>visual hacks

We were discussing "tricking" the computer vision component of a self-driving car, not getting software access. That's still a concern, but it's an entirely different set of security requirements that we already face in planes and existing cars.

Except XKCD wasn't.

To me,

>painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway

sounds like

>"tricking" the computer vision component of a self-driving car, not getting software access.

Ok, fair point about the hacking, but unintentionally misleading things are also hard to estimate.

This does not take the issue off the table, as far as practical applications of neural-network object recognition and scene analysis are concerned. Firstly, it leaves open the issue of an additional category of images for which neural networks alone very confidently misidentify things. Secondly, the time limitation, while being a perfectly valid aspect of the experiment, means that parity has not yet been achieved in these cases, either.

No

It doesn't decrease the actual set of attacks you can run against a DNN, so no. If anything it increases the scope of the security vulnerability.

I'm looking forward to when these are integrated into Recaptcha...

Aren’t we all time-limited humans?

On a long enough timeline the survival rate for everyone drops to zero.