about
LOC of Import, Wrangle, Explore, Model and Evaluate in 118k Jupyter Notebooks (arxiv.org)
1 point by breck on Sep 5, 2020 | hide | past | pdf | discuss on HN

In plain words: It reads each notebook cell's code structure and nearby comments to label it as an analysis stage, learning from cheap automatic labels instead of expert-written rules. It beat expert heuristics by 38% in accuracy, then mapped patterns across scientific notebooks.

Abstract · CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis

Large scale analysis of source code, and in particular scientific source code, holds the promise of better understanding the data science process, identifying analytical best practices, and providing insights to the builders of scientific toolkits. However, large corpora have remained unanalyzed in depth, as descriptive labels are absent and require expert domain knowledge to generate. We propose a novel weakly supervised transformer-based architecture for computing joint representations of code from both abstract syntax trees and surrounding natural language comments. We then evaluate the model on a new classification task for labeling computational notebook cells as stages in the data analysis process from data import to wrangling, exploration, modeling, and evaluation. We show that our model, leveraging only easily-available weak supervision, achieves a 38% increase in accuracy over expert-supplied heuristics and outperforms a suite of baselines. Our model enables us to examine a set of 118,000 Jupyter Notebooks to uncover common data analysis patterns. Focusing on notebooks with relationships to academic articles, we conduct the largest ever study of scientific code and find that notebook composition correlates with the citation count of corresponding papers.

Ge Zhang, Mike A. Merrill, Yang Liu, Jeffrey Heer, Tim Althoff
arXiv:2008.12828 · cs.LG, cs.DL, stat.ML · submitted Aug 28, 2020
abstract · pdf · html

add comment on HN