ML Newsnew | past | best | rss | submit about
Towards Learning to Play Piano with Dexterous Hands and Touch (arxiv.org)
39 points by ofou on Oct 5, 2021 | hide | past | pdf | 4 comments on HN

In plain words: A simulated robot hand learns piano from scratch by trial and error, rewarded for key touches as tasks grow harder step by step. Unlike usual robots with special hands and fixed plans, it finds the right keys and handles rhythm, loudness, and fingering.

Abstract

The virtuoso plays the piano with passion, poetry and extraordinary technical ability. As Liszt said (a virtuoso)must call up scent and blossom, and breathe the breath of life. The strongest robots that can play a piano are based on a combination of specialized robot hands/piano and hardcoded planning algorithms. In contrast to that, in this paper, we demonstrate how an agent can learn directly from machine-readable music score to play the piano with dexterous hands on a simulated piano using reinforcement learning (RL) from scratch. We demonstrate the RL agents can not only find the correct key position but also deal with various rhythmic, volume and fingering, requirements. We achieve this by using a touch-augmented reward and a novel curriculum of tasks. We conclude by carefully studying the important aspects to enable such learning algorithms and that can potentially shed light on future research in this direction.

Huazhe Xu, Yuping Luo, Shaoxiong Wang, Trevor Darrell, Roberto Calandra
arXiv:2106.02040 · cs.RO, cs.AI, stat.ML · submitted Jun 3, 2021 · updated Aug 5, 2022
abstract · pdf · html

add comment on HN

kgarten on Oct 6, 2021 [–]
Damn .. Hoped to find sth. About motor skill learning in the paper.

On a tangent, huge fan of passive haptic learning and the use of Soft robotics for learning.

https://www.vogue.cs.titech.ac.jp/projects/digitalsports/rob...

https://www.gvu.gatech.edu/research/projects/passive-haptic-...

Shameless self plug: we used artificial muscles to improve beginner percussion training

https://kaikunze.de/papers/pdf/goto2020accelerating.pdf

reply
CyberDildonics on Oct 5, 2021 [–]
I think this is about robotics, but who is editing these titles? I don't understand why no one would step in and clean up the nonsense instead of letting obviously wrong direct translation stand.
reply
zuminator on Oct 6, 2021 | parent [–]
What do you mean? That's the name of the actual paper, which is written in standard native-quality English. Here's the concluding paragraph:

"In this paper, we proposed the first reinforcement learning based approach for learning to play piano with robot hands equipped with tactile sensors. Since the task is new to reinforcement learning, we formalize the task as MDP and detail about specific designs for all the components of the MDP. With off-the-shelf reinforcement learning algorithms, we can train a robot hand to play the piano with correct notes, velocity and fingering. We also carefully study core factors in the whole system and provide useful information for future research on this track." [MDP = Markov Decision Process]

reply
ghusbands on Oct 6, 2021 | parent [–]
This quoted part is not native-quality English: "as MDP and detail about specific designs"

There are plenty of other examples in the paper. I would assume it has been automatically translated and then edited for clarity, though it could just be written by someone for whom English is a secondary language.

reply
About | RSS | RSS (all) | HN arXiv