about
Show HN: Intrusion Detection in Real-time (arxiv.org)
69 points by siddhartb_ on Apr 22, 2020 | hide | past | pdf | 11 comments on HN

In plain words: It watches a live stream of graph edges and flags sudden bursts of many similar edges, like coordinated attacks, instead of judging each edge alone. It runs in fixed time and memory per edge and beats the best prior tools by 42%–48% in accuracy.

Abstract · MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams

Given a stream of graph edges from a dynamic graph, how can we assign anomaly scores to edges in an online manner, for the purpose of detecting unusual behavior, using constant time and memory? Existing approaches aim to detect individually surprising edges. In this work, we propose MIDAS, which focuses on detecting microcluster anomalies, or suddenly arriving groups of suspiciously similar edges, such as lockstep behavior, including denial of service attacks in network traffic data. MIDAS has the following properties: (a) it detects microcluster anomalies while providing theoretical guarantees about its false positive probability; (b) it is online, thus processing each edge in constant time and constant memory, and also processes the data 162-644 times faster than state-of-the-art approaches; (c) it provides 42%-48% higher accuracy (in terms of AUC) than state-of-the-art approaches.

Siddharth Bhatia, Bryan Hooi, Minji Yoon, Kijung Shin, Christos Faloutsos
arXiv:1911.04464 · cs.LG, cs.AI · submitted Nov 11, 2019 · updated Aug 23, 2020
abstract · pdf · html · 8 pages, Accepted at AAAI Conference on Artificial Intelligence (AAAI), 2020 [oral paper]; minor fixes, updated experiments

add comment on HN

Code is available in C++, Python, Ruby, R, and Rust at https://github.com/bhatiasiddharth/MIDAS
Really cool project. I'm doing a real-time version (gets input on stdin, sends output on stdout), will probably release the code soon.
Can this detect DDoS like attacks also?
We handle locality in terms of both source and destination, therefore we should be able to handle both DoS and DDoS attacks.
Looks interesting. Better than some of the static detection algorithms out there
can you list out use cases where we can use this algorithm?
In addition to detecting intrusions, it can detect fake ratings and frauds. Basically finding anomalous and suspicious behavior in any dynamic (time-evolving) graph.
Hi, how will the performance be affected if let's say time ticks are not uniform?
Great question, it will be interesting to try it out. Temporal relations should be affected a bit but MIDAS should be able to detect anomalies.
Are you taking context into consideration? For example, Donald Trump's tweets will be much higher than an ordinary person's.
Yes, we take expected count of a particular user/source node into consideration.