about
Exascale Training of Generative Model with Historical Priors for Data Reduction (arxiv.org)
1 point by rbanffy 144 days ago | hide | past | pdf | discuss on HN

In plain words: A compression model trained on years of satellite images shrinks Earth observation data while keeping it usable for later scientific tasks. Because the planet is imaged repeatedly, it cuts data by up to 10,000 times, far beyond ordinary image compression.

Abstract · Transforming the Use of Earth Observation Data: Exascale Training of a Generative Compression Model with Historical Priors for up to 10,000x Data Reduction

Earth observation is becoming one of the largest data-producing activities in science, yet current pipelines still treat compression as a storage and transmission tool rather than a new way to use data. We present a generative compression framework that learns from historical Earth observation archives and enables on-demand 100x to 10,000x data reduction across downstream tasks. Unlike general visual data, Earth observation repeatedly measures the same evolving planet, making historical-prior learning feasible for extreme compression. To realize this paradigm, we train large generative compression models at exascale on the LineShine Armv9 CPU supercomputer, with co-optimization across model design, kernels, memory hierarchy, runtime, and parallelism. Our implementation sustains 1.54 EFLOP/s and peaks at 2.16 EFLOP/s in end-to-end training. This work shows that historical-prior generative compression can turn Earth observation data into an active, task-adaptive foundation for acquisition, delivery, storage, and scientific use.

Jinxiao Zhang, Runmin Dong, Xiyong Wu, Xihan Huang, Shenggan Cheng, Yunkai Yang, Zheng Zhou, Yunpu Xu, Zhaoyang Luo, Miao Yang, Fan Wei, Mengxuan Chen, et al.
arXiv:2605.08633 · cs.DC, cs.CV · submitted May 9, 2026
abstract · pdf · html

add comment on HN