about
EXIF as Language: Cross-Modal Associations Between Images and Camera Metadata (arxiv.org)
3 points by tosh on Jan 13, 2023 | hide | past | pdf | discuss on HN

In plain words: The system trains image patches to line up with the camera settings stored in a photo's file, written as plain text, so it learns what different cameras look like. It beat self-trained and human-labeled features on forensics tasks, spotting edited regions without extra training.

Abstract · EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata

We learn a visual representation that captures information about the camera that recorded a given photo. To do this, we train a multimodal embedding between image patches and the EXIF metadata that cameras automatically insert into image files. Our model represents this metadata by simply converting it to text and then processing it with a transformer. The features that we learn significantly outperform other self-supervised and supervised features on downstream image forensics and calibration tasks. In particular, we successfully localize spliced image regions "zero shot" by clustering the visual embeddings for all of the patches within an image.

Chenhao Zheng, Ayush Shrivastava, Andrew Owens
arXiv:2301.04647 · cs.CV, cs.CL · submitted Jan 11, 2023 · updated Jun 17, 2023
abstract · pdf · html · CVPR 2023 (Highlight). Project link: http://hellomuffin.github.io/exif-as-language

add comment on HN