In plain words: Instead of editing a robot's sensor readings, an attacker simply plays a strange opponent whose moves create confusing situations. Such opponents reliably beat top self-play-trained humanoid robots while looking random, and work better in more complex arenas.
Abstract
Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, Stuart Russell
arXiv:1905.10615 · cs.LG, cs.AI, cs.CR, stat.ML · submitted May 25, 2019 · updated Jan 17, 2021
abstract · pdf · html · Presented at ICLR 2020
“In some ways, adversarial policies are more worrying than attacks on supervised learning models, because reinforcement learning policies govern an AI’s overall behavior.If a driverless car misclassifies input from its camera, it could fall back on other sensors, for example.” TIL fail-safe components are 1) ubiquitous 2) work 3) only an option for supervised learning components.
“A supervised learning model, trained to classify images, say, is tested on a different data set from the one it was trained on to ensure that it has not simply memorized a particular bunch of images. But with reinforcement learning, models are typically trained and tested in the same environment.” First, a RL environment is not equivalent to a supervised learning data set. Second, the train validate test paradigm is not thrown out in RL research, its why OpenAI put their Starcraft agent on public ladders.
“The good news is that adversarial policies may be easier to defend against than other adversarial attacks.” This sentence refers to Graves et al. adversarially training their agents. Adversarial training is, of course, also conducted frequently in supervised learning.