In plain words: A helper trained on a little labeled gameplay guesses which keys and mouse moves a player pressed, then labels piles of unlabeled Minecraft videos to teach an agent to act. It mastered hard tasks impossible to learn from scratch, and first crafted diamond tools.
Abstract
Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision domains such as robotics, video games, and computer use, publicly available data does not contain the labels required to train behavioral priors in the same way. We extend the internet-scale pretraining paradigm to sequential decision domains through semi-supervised imitation learning wherein agents learn to act by watching online unlabeled videos. Specifically, we show that with a small amount of labeled data we can train an inverse dynamics model accurate enough to label a huge unlabeled source of online data -- here, online videos of people playing Minecraft -- from which we can then train a general behavioral prior. Despite using the native human interface (mouse and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-shot capabilities and that it can be fine-tuned, with both imitation learning and reinforcement learning, to hard-exploration tasks that are impossible to learn from scratch via reinforcement learning. For many tasks our models exhibit human-level performance, and we are the first to report computer agents that can craft diamond tools, which can take proficient humans upwards of 20 minutes (24,000 environment actions) of gameplay to accomplish.
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune
arXiv:2206.11795 · cs.LG, cs.AI · submitted Jun 23, 2022
abstract · pdf · html
A while ago, we participated in the Bomberland competition (won semi-finals, 2nd place overall) and we went with deep learning reinforcement learning and here's the rough math: It takes 1 step to place the bomb, 5 for the bomb to arm, 1 to detonate, 10 while there's fire. So you might not know until 17 steps later if placing that bomb was a good idea. You have 7 actions (up,down,left,right,bomb,detonate,nop) so the chance of getting that 17-step sequence correct by chance is (1/7)^17 = 4*10^-15. Or said the other way around, you need to let the AI try around 232,630,513,987,207 times to get it correct once.
And that's why imitating someone else - no matter if they are an expert or not - will massively boost your learning performance. Because even the worst Mindcraft player on YouTube is still doing 1000x better than truly random exploration.
(That said, we lost in the Bomberland finals against someone who analyzed the game theoretically and then just hard-coded the perfect strategy. Sometimes thinking hard is superior to all AI approaches...)