about
Deep-Spying: Spying Using Smartwatch and Deep Learning (arxiv.org)
84 points by kushti on Dec 18, 2015 | hide | past | pdf | 18 comments on HN

In plain words: A neural network that tracks sequences reads a smartwatch's motion sensors to guess which of 12 keypad keys a person taps. It worked with above-average accuracy on raw, unprocessed readings, skipping the careful cleanup and hand-picked measurements such attacks normally need.

Abstract · Deep-Spying: Spying using Smartwatch and Deep Learning

Wearable technologies are today on the rise, becoming more common and broadly available to mainstream users. In fact, wristband and armband devices such as smartwatches and fitness trackers already took an important place in the consumer electronics market and are becoming ubiquitous. By their very nature of being wearable, these devices, however, provide a new pervasive attack surface threatening users privacy, among others. In the meantime, advances in machine learning are providing unprecedented possibilities to process complex data efficiently. Allowing patterns to emerge from high dimensional unavoidably noisy data. The goal of this work is to raise awareness about the potential risks related to motion sensors built-in wearable devices and to demonstrate abuse opportunities leveraged by advanced neural network architectures. The LSTM-based implementation presented in this research can perform touchlogging and keylogging on 12-keys keypads with above-average accuracy even when confronted with raw unprocessed data. Thus demonstrating that deep neural networks are capable of making keystroke inference attacks based on motion sensors easier to achieve by removing the need for non-trivial pre-processing pipelines and carefully engineered feature extraction strategies. Our results suggest that the complete technological ecosystem of a user can be compromised when a wearable wristband device is worn.

Tony Beltramelli, Sebastian Risi
arXiv:1512.05616 · cs.CR, cs.CY, cs.LG · submitted Dec 17, 2015
abstract · pdf · html · Security, Side-Channel Attack, Keystroke Inference, Motion Sensors, Deep Learning, Recurrent Neural Network, Wearable Computing

add comment on HN
Also discussed: Mar 2020 (2 points, 0 comments)

Moneyquote of the summary: "Our results suggest that the complete technological ecosystem of a user can be compromised when a wearable wristband device is worn."
Summary:using the motion sensors in the smartwatch, they can figure out which keys you are pressing, and thus read your passwords.
I believe a recent episode of the TV series "Minority Report" used this as part of the story.
Hey guys! :)

The source code is available at: https://github.com/tonybeltramelli/Deep-Spying

And a video demo at: https://youtu.be/ZBwSfvnoq5U

A very interesting caveat from the paper:

"It is important to note that it is assumed that the victim is wearing the WAD on the wrist of the preferred hand used to interact with keyboards. In fact, in our attack scenario it would be harder, if not impossible, for the attacker to infer keystrokes if the victim is right-handed but is wearing the WAD on the left hand for example."

A WAD is any wearable arm band device.

Which is how most people.wear watches (especially expensive ones).
I don't and none of my friends seem to either.
That's probably a cultural thing. In my country, most people (myself included) wear watches on non-dominant hand.
The reason one would want to do this is to reduce wear on the watch, since the dominant hand is going and doing things the dominant wrist is more likely to be smacked. This goes double for expensive watches.
Personally, I think it's the opposite. I wear my watch on the wrist of my dominant hand. I've never had a problem; in fact I think you are more likely to pay attention to it and not smack it into things.
I am the only right handed person that I know who wears their watch on their right hand. Most people think I am left handed because of it.

I think I started it to be different when I was a teen and the habit just stuck.

I am right-handed and wear my watch on my right wrist as well. I've never ran into any problems doing so, and manipulating buttons on different watches was never a problem either.
You wear your watch on your dominant hand? That feels very unnatural to me. Just manipulating the straps alone seems difficult, so that's a barrier right there...
Not to forget, that most Watches are designed to be worn on the left side. (Position of the Buttons etc.) Smartwatches are improving here, in displaying a rotated screen, but often even these are asymmetrical designed and thus harder to use on the right arm.
> most Watches are designed to be worn on the left side.

Yes, for right-handed people. The opposite is true for left-handed people, and left-handed versions of watches.

I do as well. Even since I was a kid, wearing watches on the wrist of my non-dominant hand felt unnatural. I have no problem manipulating the strap or buttons on my watches.
Before everyone panics, there's an important note on page 70:

>The LSTM model can also successfully classify signals with an accuracy of 19% when the dataset used for training and logging are originated from two different keypads.

Still scary to think how a motivated team could extend this though.

There's an important conceptual link between training RNNs and side channel attacks.

I wrote a bit on the topic. Unfortunately, not my finest writing in terms of clarity, but I think it touches something important.

https://medium.com/@arthurb/traces-probability-and-learning-...