ML Newsnew | past | best | rss | submit about
On Extractive and Abstractive Summarization with Transformer Language Models (arxiv.org)
106 points by hirundo on Sep 10, 2019 | hide | past | pdf | 12 comments on HN

In plain words: First it picks out the most important sentences from a long document, then feeds them to a language model that writes a fresh summary in its own words. This beats the usual trick of copying phrases straight from the text, scoring higher on the standard summary-rating scale while paraphrasing more.

Abstract · On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We show that this extractive step significantly improves summarization results. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher rouge scores. Note: The abstract above was not written by the authors, it was generated by one of the models presented in this paper.

Sandeep Subramanian, Raymond Li, Jonathan Pilault, Christopher Pal
arXiv:1909.03186 · cs.CL · submitted Sep 7, 2019 · updated Apr 28, 2020
abstract · pdf · html

add comment on HN

mayank on Sep 10, 2019 [–]
It's definitely cute that the abstract was generated by the model, but I wouldn't give that too much weight because it's the definition of cherry-picking. In this case, you can pick your data (the contents of the paper) to match a desirable output from the model (the abstract).
reply
ma2rten on Sep 10, 2019 | parent [–]
I think it would be a lot of effort to keep changing the paper until you get the perfect abstract. It would be easier to train different models or do random sampling from the predicted distribution.
reply
semi-extrinsic on Sep 10, 2019 [–]
This is impressive. They trained it on 200k articles from arXiv, 130k from PubMed, and over 1 million each from Newsroom and BigPatent. They have comparisons of generated abstracts versus actual abstracts of some landmark NLP papers.

My only gripe is that I would have liked to see (maybe in an Appendix) examples on papers on completely different topics, say one in biology, one in math and one in physics. It would be difficult to pick good examples, sure. But it would significantly strengthen at least my impression of the transferability.

reply
jcims on Sep 11, 2019 | parent [–]
Would be interesting to see what kind of paper it writes.
reply
aantix on Sep 10, 2019 [–]
I really wish that the code or at least a working demo was a requirement to make such claims public.
reply
tastroder on Sep 11, 2019 | parent [–]
Playing the devil's advocate here: might I ask why? The paper seems pretty thorough w.r.t. the description of used corpora, models, and hyperparameters. They even point to the exact implementation of their evaluation scoring and include a few examples in the paper itself. Even if they put up a demo instance for the required infrastructure it would be dead as soon as it hit HN and, as research code goes, likely a security hazard to wherever it's hosted.

In my view there seems to be enough here to replicate and validate the claims yourself if you wanted to. With a basic level of trust in academic integrity I'm completely fine with this paper.

reply
rajangdavis on Sep 11, 2019 | parent [–]
From how they set up the training, I think this is a nontrivial task. Also, from a casual read through, it looks like it is generally focused on Arxiv papers.

To their credit, the authors included the models used and the metrics they used to validate their model. They also have detailed notes on the architecture for training which, at a quick glance, doesn't look easy to replicate unless you can borrow some GPU's in the cloud.

reply
RobertDeNiro on Sep 12, 2019 | parent [–]
It focused on arxiv because you need a large set of labelled data (i.e. long documents with summaries). There is not many datasets of that kind out there.
reply
JonathanFly on Sep 11, 2019 [–]
Imagine a future where 'this abstract was generated by the model' is in the training material for future papers https://twitter.com/jonathanfly/status/1171551688668471297
reply
kieckerjan on Sep 10, 2019 [–]
Now if that is true, that is one badass summary!
reply
dang on Sep 10, 2019 | parent [–]
(The submitted title was "This abstract was generated by one of the models presented in this paper".)
reply
nullsmack on Sep 10, 2019 [–]
abstraception
reply
About | RSS | RSS (all) | HN arXiv