ML Newsnew | past | best | rss | submit about
Surpassing Human-Level Performance on ImageNet Classification [pdf] (arxiv.org)
53 points by fchollet on Feb 9, 2015 | hide | past | pdf | 13 comments on HN

In plain words: Instead of always clamping negative signals to zero, this network learns how much of each negative to keep, and starts with carefully chosen weights so very deep versions train from scratch. It made 4.94% errors on ImageNet, beating both the previous best system and human accuracy.

Abstract · Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
arXiv:1502.01852 · cs.CV, cs.AI, cs.LG · submitted Feb 6, 2015
abstract · pdf · html

add comment on HN
Also discussed: Feb 2015 (9 points, 0 comments)
Animats on Feb 10, 2015 [–]
That's an impressive result, considering how simple the algorithm really is. (The learning algorithm isn't obvious, but it's not a lot of code.)

Can this algorithm be run in reverse, to generate an image from the network? That's been done with one of the other deep neural net classifiers. "School bus" came out as a yellow blob with horizontal black lines.

An interesting question is whether there's a bias in the data set because humans composed the pictures, and humans like to take pictures of certain things. (Cats are probably over-represented.) Images taken by humans tend to have a primary subject, and that subject is usually roughly centered in the images. It might be useful to test against a data set taken from Google StreetView images, which lack such a composition bias.

reply
dEnigma on Feb 10, 2015 | parent [–]
Can this algorithm be run in reverse, to generate an image from the network? That's been done with one of the other deep neural net classifiers. "School bus" came out as a yellow blob with horizontal black lines.

Do you have a link for that, sounds interesting (nothing turned up in a quick google search)

edit: I found something similar to what you were talking about, is this[1] what you meant?

[1]http://www.evolvingai.org/fooling

reply
Houshalter on Feb 10, 2015 | parent [–]
That's the paper he's referring to. They used another NN to generate images, and selected the ones that the first NN predicted to be school buses the most.
reply
svantana on Feb 9, 2015 [–]
Interesting, I've long wondered why parametric nonlinearities are not used more often. It adds very little to the overall parameter count (the number of units is most often dwarfed by the number of connections), but it should increase expressibility a lot (e.g. adaptively using soft or hard nonlinearities). Taken to its extreme, I've been toying with the idea of combining DNNs and Genetic Programming - using a large number of arbitrary symbolic expressions with high connectivity and many layers.
reply
fchollet on Feb 9, 2015 | parent [–]
Genetic programming is an inefficient search method, and will require many evaluations of the cost function to optimize anything. In the case of DCNNs, evaluating an architecture can keep a modern GPU busy for days, so genetic algorithms are pretty much out of the question.

I think an easy way to improve our models in the short term is to make more of the parameters we use be learnable: the parameters of the non-linearity are a good place to start with, and another would be the parameters of the data augmentation transformations.

One could consider that learned data augmentation schemes implement a form of guided visual attention.

reply
dwiel on Feb 10, 2015 [–]
It seems that this might mean that ImageNet is becoming less useful a benchmark dataset. Some of the images have labels which I would have never guessed myself, and not because I don't know the difference between two types of stingray, but because I would have never said that the topic of the image was a seatbelt. It would be interesting to know how many errors are actually due to outputting a label which does exist in the image, but isn't in the labeled truth data and how many are plainly not in the image.
reply
fchollet on Feb 10, 2015 | parent [–]
The accuracy reported is top-5 accuracy, meaning that a model is considered correct on a test image if it includes the expected label in its top 5 predicted labels. This does mitigate the multi-object issue quite a bit.
reply
dwiel on Feb 10, 2015 | parent [–]
Most of the examples cited in the paper that their algorithm got wrong, I would also get 'wrong.' I don't think I would guess restaurant for any of the images with that label, I might have gotten the middle spotlight and I might have gotten the first letter opener right, but I'm not sure.

How do you explain that their performance is better than human? Is it in the obscure examples?

reply
greeneggs on Feb 10, 2015 | parent [–]
On the other hand, some of their correct images have questionable captions as well. For example, they labeled the geyser picture correctly, but their top labels also included "sandbar", "breakwater" and "leatherback turtle". A better scoring function, perhaps including hierarchies to account for the very vague "restaurant" photos and very specific dog breed photos, might be helpful. Otherwise, it seems like we might be overfitting to the peculiarities of this dataset.
reply
AndrewKemendo on Feb 10, 2015 | parent [–]
Interesting. I typically am annoyed when people move the goalposts like this on AI but in my research it seems like the ImageNet library is annotating recursively through the various automation sets and not actually founded on human classification.

Is that accurate?

reply
dwiel on Feb 10, 2015 | parent [–]
That is a good question. I went and read the about page at imagenet [1], but it wasn't completely obvious what the source is. One thing to note though is that the context as described there is "1000 images to illustrate each word." This doesn't exactly match with: "words which a person would most likely label an image." Check out the page on team sport for example [2]. Would you ever label one of those team sport? Or what about this one labeled 'fresh bean' [3]. It is not hard to find hard examples here [4].

  [1]: http://www.image-net.org/about-overview
  [2]: http://www.image-net.org/synset?wnid=n00887544
  [3]: http://www.image-net.org/nodes/8/07727578/5b/5bcde1a26fd46c485ca7df589e57ba2ded9cb789.thumb
  [4]: http://www.image-net.org/synset
reply
AndrewKemendo on Feb 10, 2015 | parent [–]
I asked the question because it seems like their library has too many images for manual input and I never found a good answer on what the seed set was on the site.

I don't think the question should be "would you label these..." but moreso "is description 'X' an accurate representation of the image." In the latter case I say yes to most of the examples I find. The former question severely limits the set of possibilities to whomever is the one classifying it.

I know there has been a ton of research done on this stuff but to me the context threshold for image classification seems difficult to bound because humans might be looking for something very specific in a photo, to the point of impossibly difficult to annotate.

For example take this photo: http://farm1.static.flickr.com/222/497812075_d412946eef.jpg

I can think of a million annotations for it: - 9 men - Island - Trees near water - Rocks in trees - Backwards hat - Blue water - Rowing - Oar splashing - Nice day - Warm weather - White boat - Saddleback mountain etc...

So how would you rank those in terms of what the picture is really showing? The only thing I can think is of popularity, which makes the most sense to me in terms of how we natively classify - but it doesn't cover boundary cases or super narrow specificity.

reply
dwiel on Feb 10, 2015 | parent [–]
Looks like they use Amazon Turk: http://www.image-net.org/papers/imagenet_cvpr09.pdf

Yeah that's a good point that "is description X and accurate representation" is a better question, though harder to ground truth on since the truth is a large set of possible 'accurate representations.'

From the PReLU paper, I found this:

> Russakovsky et al. [22] recently reported that human performance yields a 5.1% top-5 error on the ImageNet dataset. This number is achieved by a human annotator who is well trained on the validation images to be better aware of the existence of relevant classes. When annotating the test images, the human annotator is given a special interface, where each class title is accompanied by a row of 13 example training images. The reported human performance is estimated on a random subset of 1500 test images.

reply
About | RSS | RSS (all) | HN arXiv